09/01/2025 01:56:34.148 7f5ef8fbef40 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:56:34.148 7f5ef8fbef40 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:56:34.148 7f5ef8fbef40 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:56:34.148 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.249:2181,10.0.0.38:2181,10.0.0.131:2181 09/01/2025 01:56:34.149 7f5ef8fbef40 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:56:34.150 7f5ef4045900 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:56:34.150 7f5ef4045900 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:56:34.151 7f5ef4045cc0 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:56:34.152 7f5ef4045cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.153 7f5ef4045cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:56:34.154 7f5ef4045cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.167 7f5ef4045cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.167 7f5ef4045cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:34.167 7f5ef4045cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.267 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.249:9041', '10.0.0.38:9041', '10.0.0.131:9041'] 09/01/2025 01:56:34.267 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:56:34.267 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:56:34.268 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:56:34.268 7f5ef4045d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.268 7f5ef4045d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:34.268 7f5ef4045d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.271 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:56:34.357 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:56:34.358 7f5ef4c1c5e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.358 7f5ef4c1c5e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:34.358 7f5ef4c1c5e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.367 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:56:34.368 7f5ef4c1c2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.368 7f5ef4c1c2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:34.368 7f5ef4c1c2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.565 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:35.569 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid', created 09/01/2025 01:56:35.574 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:37.477 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:38.152 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:38.153 7f5ef4b2aea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:38.153 7f5ef4b2aea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:38.153 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:38.153 7f5ef4b2aea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:38.154 7f5ef4b2aea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.602 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 01:56:38.607 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:40.495 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:41.587 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent', created 09/01/2025 01:56:41.592 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:42.154 7f5ef4b532c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:42.155 7f5ef4b532c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:42.155 7f5ef4b532c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:42.155 7f5ef4b532c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:42.155 7f5ef4b532c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:42.155 7f5ef4b532c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:42.155 7f5ef4b532c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:42.156 7f5ef4b532c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:44.227 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:44.690 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 01:56:44.694 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:44.695 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:56:44.732 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:56:46.155 7f5ef4bd4e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:46.156 7f5ef4bd4e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:46.156 7f5ef4bd4e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:46.156 7f5ef4bd4e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:46.156 7f5ef4bd4e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:46.157 7f5ef4bd4e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.589 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:47.537 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table', created 09/01/2025 01:56:47.547 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:48.705 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:49.453 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:56:49.458 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:50.157 7f5ef4090720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:50.159 7f5ef4090720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:50.159 7f5ef4090720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:50.160 7f5ef4090720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:50.160 7f5ef4090720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:50.160 7f5ef4090720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:50.160 7f5ef4090720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:50.160 7f5ef4090720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:50.161 7f5ef4090720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.161 7f5ef4090720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:50.161 7f5ef4090720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.161 7f5ef4090720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.161 7f5ef4090720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:50.161 7f5ef4090720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:51.512 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:52.653 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table', created 09/01/2025 01:56:52.660 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:53.603 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 09/01/2025 01:56:54.159 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:54.160 7f5ef4b2aea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:54.160 7f5ef4b2aea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.161 7f5ef4b2aea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:54.162 7f5ef4b2aea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.527 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 01:56:54.531 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 09/01/2025 01:56:54.532 7f5eefde2400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.532 7f5eefde2400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.532 7f5eefde2400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.604 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:56:54.612 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:56:54.613 7f5eeffeb720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.613 7f5eeffeb720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.613 7f5eeffeb720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.613 7f5eeffeb720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.613 7f5eeffeb720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.613 7f5eeffeb720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.617 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:56:54.617 7f5eeffeb720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.618 7f5eeffeb720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.618 7f5eeffeb720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.648 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:54.648 7f5eeffb10e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.648 7f5eeffb10e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.648 7f5eeffb10e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.651 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:54.652 7f5eeffb10e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.652 7f5eeffb10e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.652 7f5eeffb10e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.653 7f5eeffb1400 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:54.653 7f5eeffb1680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.653 7f5eeffb1680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.653 7f5eeffb1680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.664 7f5eeffb1400 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:54.664 7f5eeffb1900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.665 7f5eeffb1900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.665 7f5eeffb1900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.743 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config']: Unknown id: global_system_config default-global-system-config 09/01/2025 01:56:54.902 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-vrouter-config']: Unknown id: global_vrouter_config default-global-system-config:default-global-vrouter-config 09/01/2025 01:56:54.976 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain']: Unknown id: domain default-domain 09/01/2025 01:56:55.337 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management', 'default-application-policy-set']: Unknown id: application_policy_set default-policy-management:default-application-policy-set 09/01/2025 01:56:55.390 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric']: Unknown id: virtual_network default-domain:default-project:ip-fabric 09/01/2025 01:56:55.484 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric']: Unknown id: routing_instance default-domain:default-project:ip-fabric:ip-fabric 09/01/2025 01:56:55.543 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', '__default__']: Unknown id: routing_instance default-domain:default-project:ip-fabric:__default__ 09/01/2025 01:56:55.623 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__']: Unknown id: virtual_network default-domain:default-project:__link_local__ 09/01/2025 01:56:55.828 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__', '__link_local__']: Unknown id: routing_instance default-domain:default-project:__link_local__:__link_local__ 09/01/2025 01:56:55.933 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network']: Unknown id: virtual_network default-domain:default-project:dci-network 09/01/2025 01:56:56.011 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', 'dci-network']: Unknown id: routing_instance default-domain:default-project:dci-network:dci-network 09/01/2025 01:56:56.057 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', '__default__']: Unknown id: routing_instance default-domain:default-project:dci-network:__default__ 09/01/2025 01:56:56.095 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'system-defined-bottle-request-size-limit']: Unknown id: alarm default-global-system-config:system-defined-bottle-request-size-limit 09/01/2025 01:56:56.141 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network']: Unknown id: routing_instance default-domain:default-project:default-virtual-network:default-virtual-network 09/01/2025 01:56:56.194 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__no_rule__']: Unknown id: security_group default-domain:default-project:__no_rule__ 09/01/2025 01:56:56.282 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-discovery-service-assignment']: Unknown id: discovery_service_assignment default-discovery-service-assignment 09/01/2025 01:56:56.316 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-qos-config']: Unknown id: global_qos_config default-global-system-config:default-global-qos-config 09/01/2025 01:56:56.351 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'service-chain-flat-ipam']: Unknown id: network_ipam default-domain:default-project:service-chain-flat-ipam 09/01/2025 01:56:56.395 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['label']: Unknown id: tag_type label 09/01/2025 01:56:56.432 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['application']: Unknown id: tag_type application 09/01/2025 01:56:56.481 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['tier']: Unknown id: tag_type tier 09/01/2025 01:56:56.507 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['deployment']: Unknown id: tag_type deployment 09/01/2025 01:56:56.529 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['site']: Unknown id: tag_type site 09/01/2025 01:56:56.551 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_fwaas']: Unknown id: tag_type neutron_fwaas 09/01/2025 01:56:56.584 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_tag']: Unknown id: tag_type neutron_tag 09/01/2025 01:56:56.616 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.131_0 09/01/2025 01:56:56.616 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly) 09/01/2025 01:56:56.655 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (None) 09/01/2025 01:56:56.655 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running dbe-resync 09/01/2025 01:56:56.671 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type tag_type len 7 09/01/2025 01:56:56.672 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for tag_type 09/01/2025 01:56:56.685 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Error in _create_default_rbac_rule: NoIdError: Unknown id: api_access_list default-global-system-config:default-api-access-list 09/01/2025 01:56:56.686 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for tag_type 09/01/2025 01:56:56.687 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type routing_instance len 6 09/01/2025 01:56:56.687 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for routing_instance 09/01/2025 01:56:56.688 7f5ef8fbef40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-api-access-list']: Unknown id: api_access_list default-global-system-config:default-api-access-list 09/01/2025 01:56:56.700 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for routing_instance 09/01/2025 01:56:56.700 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type policy_management len 1 09/01/2025 01:56:56.700 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for policy_management 09/01/2025 01:56:56.703 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for policy_management 09/01/2025 01:56:56.703 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type virtual_network len 4 09/01/2025 01:56:56.704 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for virtual_network 09/01/2025 01:56:56.769 7f5eeffb1ea0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:56.771 7f5eef5bc040 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:56.771 7f5eef5bc180 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:56.772 7f5eef5bc2c0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 09/01/2025 01:56:56.772 7f5eef5bc400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.772 7f5eef5bc400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:ip-fabric elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "ip-fabric"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11906273895913767761, "uuid_lslong": 12936707290311336679}, "enable": true, "created": "2025-09-01T01:56:55.457028", "last_modified": "2025-09-01T01:56:56.739049", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : trueis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "a53b9506-b472-4351-b388-6abd2a36a2e7"virtual_network_category : "tenant"virtual_network_network_id : 2 } deleted = False >> 09/01/2025 01:56:56.772 7f5eef5bc400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.772 7f5eef5bc400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:dci-network elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "dci-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10402109786093929934, "uuid_lslong": 11584685736362511906}, "enable": true, "created": "2025-09-01T01:56:55.995390", "last_modified": "2025-09-01T01:56:56.746574", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falseis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "905bb7bc-dabe-45ce-a0c5-124c866c0e22"virtual_network_category : "tenant"virtual_network_network_id : 4 } deleted = False >> 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:__link_local__ elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "__link_local__"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3583347645795025512, "uuid_lslong": 9648904630498389471}, "enable": true, "created": "2025-09-01T01:56:55.815663", "last_modified": "2025-09-01T01:56:56.745243", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falseis_shared : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"parent_uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "31ba9bb2-ce58-4268-85e7-cba3ac7a6ddf"virtual_network_category : "tenant"virtual_network_network_id : 3 } deleted = False >> 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-virtual-network elements = { fq_name : ["default-domain", "default-project", "default-virtual-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18034049945446728127, "uuid_lslong": 11575957198380798807}, "enable": true, "created": "2025-09-01T01:56:55.139325", "last_modified": "2025-09-01T01:56:56.744498", "description": null, "user_visible": true, "creator": null}is_shared : falseparent_type : "project"parent_uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fa45d0e0-7c01-41bf-a0a6-0fbd689d1357"virtual_network_network_id : 1 } deleted = False >> 09/01/2025 01:56:56.773 7f5eef5bc400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.773 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for virtual_network 09/01/2025 01:56:56.773 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type network_ipam len 2 09/01/2025 01:56:56.773 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for network_ipam 09/01/2025 01:56:56.913 7f5eef3b6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.913 7f5eef3b6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:56:56.913 7f5eef3b6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.918 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting consumer greenlet 09/01/2025 01:56:56.918 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:56:56.918 7f5eef38b720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting publisher greenlet 09/01/2025 01:56:56.919 7f5eef38b720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Publisher connection 09/01/2025 01:56:56.919 7f5ef8fbef40 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-3..dummy 09/01/2025 01:56:56.919 7f5ef8fbef40 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 01:56:56.920 7f5eef3b6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.921 7f5eef3b6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ConfigApiWorkerTrace: data = << name = worker-0 worker_id = 0 introspect_port = 8084 admin_port = 8095 introspect_url = http://10.0.0.131:8084 >> 09/01/2025 01:56:56.921 7f5eef3b6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:56.921 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for network_ipam 09/01/2025 01:56:56.921 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_vrouter_config len 1 09/01/2025 01:56:56.921 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_vrouter_config 09/01/2025 01:56:56.925 7f5eef38b5e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:56:56.927 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_vrouter_config 09/01/2025 01:56:56.927 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_system_config len 1 09/01/2025 01:56:56.927 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_system_config 09/01/2025 01:56:56.928 7f5eef38b720 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 09/01/2025 01:56:56.937 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_system_config 09/01/2025 01:56:56.937 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type security_group len 1 09/01/2025 01:56:56.937 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for security_group 09/01/2025 01:56:56.941 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for security_group 09/01/2025 01:56:56.941 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type alarm len 1 09/01/2025 01:56:56.941 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for alarm 09/01/2025 01:56:56.948 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for alarm 09/01/2025 01:56:56.948 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type domain len 1 09/01/2025 01:56:56.949 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for domain 09/01/2025 01:56:56.968 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for domain 09/01/2025 01:56:56.968 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_qos_config len 1 09/01/2025 01:56:56.968 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_qos_config 09/01/2025 01:56:56.978 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_qos_config 09/01/2025 01:56:56.978 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type namespace len 1 09/01/2025 01:56:56.978 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for namespace 09/01/2025 01:56:56.982 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for namespace 09/01/2025 01:56:56.982 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type discovery_service_assignment len 1 09/01/2025 01:56:56.982 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for discovery_service_assignment 09/01/2025 01:56:56.986 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for discovery_service_assignment 09/01/2025 01:56:56.987 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type project len 1 09/01/2025 01:56:56.987 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for project 09/01/2025 01:56:56.993 7f5eeffb12c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-application-policy-set']: Unknown id: application_policy_set default-domain:default-project:default-application-policy-set 09/01/2025 01:56:57.020 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-3..dummy'] 09/01/2025 01:56:57.045 7f5eef329360 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/01/2025 01:56:57.046 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.046 7f5eef3294a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:56:55.060606", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:56:57.046 7f5eef3294a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.046 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for project 09/01/2025 01:56:57.046 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type application_policy_set len 1 09/01/2025 01:56:57.046 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for application_policy_set 09/01/2025 01:56:57.055 7f5eef38bea0 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:56:57.056 7f5eef329040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.056 7f5eef329040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:56:55.369695", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:56:57.056 7f5eef329040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.056 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for application_policy_set 09/01/2025 01:56:57.056 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type fabric len 1 09/01/2025 01:56:57.056 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for fabric 09/01/2025 01:56:57.060 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for fabric 09/01/2025 01:56:57.067 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: Completed successfully 09/01/2025 01:56:57.068 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:56:57.071 7f5ef4bc04a0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 09/01/2025 01:56:57.078 7f5eef38b5e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:56:57.095 7f5ef4bc04a0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Time elapsed in resyncing db: 0:00:00.479216 09/01/2025 01:56:57.097 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'obj_dict': {'autonomous_system': 64512, 'config_version': '1.0', 'enable_4byte_as': False, 'enable_security_policy_draft': False, 'fq_name': ['default-global-system-config'], 'ibgp_auto_mesh': True, 'id_perms': {'created': '2025-09-01T01:56:54.837560', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:54.837560', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9453937659331017976, 'uuid_mslong': 16062397226589833301}}, 'igmp_enable': False, 'mac_aging_time': 300, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'}, 'oper': 'CREATE', 'request-id': 'req-f21409d3-0f38-4727-9a04-0a1496494bbd', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:56:57.098 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'obj_dict': {'enable_security_logging': True, 'encryption_mode': 'none', 'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'global_max_vmi_flows': 0, 'id_perms': {'created': '2025-09-01T01:56:54.952618', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:54.952618', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11140519902359729818, 'uuid_mslong': 8487043090036638873}}, 'max_vm_flows_percent': 0, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'}, 'oper': 'CREATE', 'request-id': 'req-d3b26a92-67b5-4ced-9164-fdd6e0d5ffe0', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:56:57.099 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain'], 'obj_dict': {'fq_name': ['default-domain'], 'id_perms': {'created': '2025-09-01T01:56:55.001834', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.001834', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11358182388029772663, 'uuid_mslong': 5456073433214304752}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377'}, 'oper': 'CREATE', 'request-id': 'req-b472efb8-09d1-4b02-bed5-2eeb92226943', 'type': 'domain', 'uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377'} 09/01/2025 01:56:57.100 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'obj_dict': {'_enable_security_policy_draft': False, '_type': 'project', '_uuid': None, 'fq_name': ['default-domain', 'default-project'], 'id_perms': {'created': '2025-09-01T01:56:55.060606', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.060606', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13033980887668934922, 'uuid_mslong': 3180535053083298319}}, 'name': 'default-project', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'}, 'oper': 'CREATE', 'request-id': 'req-99907aae-f292-4a82-b539-96b056598e5c', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:56:57.101 7f5ef4bc04a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.101 7f5ef4bc04a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { _enable_security_policy_draft : false_type : "project"_uuid : nullfq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:56:55.060606", "description": null, "user_visible": true, "creator": null}name : "default-project"parent_type : "domain"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:56:57.101 7f5ef4bc04a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.102 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'obj_dict': {'_fabric_snat': False, '_flood_unknown_unicast': False, '_igmp_enable': False, '_is_provider_network': False, '_layer2_control_word': False, '_mac_aging_time': 300, '_mac_ip_learning_enable': False, '_mac_learning_enabled': False, '_pbb_etree_enable': False, '_pbb_evpn_enable': False, '_port_security_enabled': True, '_type': 'virtual-network', '_uuid': None, '_virtual_network_category': 'tenant', 'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'id_perms': {'created': '2025-09-01T01:56:55.139325', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.139325', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11575957198380798807, 'uuid_mslong': 18034049945446728127}}, 'name': 'default-virtual-network', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fa45d0e0-7c01-41bf-a0a6-0fbd689d1357', 'virtual_network_network_id': 1}, 'oper': 'CREATE', 'request-id': 'req-b9f055e6-fdc8-48d5-95d9-d31f8b951a75', 'type': 'virtual_network', 'uuid': 'fa45d0e0-7c01-41bf-a0a6-0fbd689d1357'} 09/01/2025 01:56:57.102 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.102 7f5eef3294a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-virtual-network elements = { _fabric_snat : false_flood_unknown_unicast : false_igmp_enable : false_is_provider_network : false_layer2_control_word : false_mac_aging_time : 300_mac_ip_learning_enable : false_mac_learning_enabled : false_pbb_etree_enable : false_pbb_evpn_enable : false_port_security_enabled : true_type : "virtual-network"_uuid : null_virtual_network_category : "tenant"fq_name : ["default-domain", "default-project", "default-virtual-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18034049945446728127, "uuid_lslong": 11575957198380798807}, "enable": true, "created": "2025-09-01T01:56:55.139325", "last_modified": "2025-09-01T01:56:55.139325", "description": null, "user_visible": true, "creator": null}name : "default-virtual-network"parent_type : "project"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fa45d0e0-7c01-41bf-a0a6-0fbd689d1357"virtual_network_network_id : 1 } deleted = False >> 09/01/2025 01:56:57.102 7f5eef3294a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.105 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'obj_dict': {'_type': 'network-ipam', '_uuid': None, 'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'id_perms': {'created': '2025-09-01T01:56:55.191109', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.191109', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10580441645665762721, 'uuid_mslong': 5519056136150665973}}, 'name': 'default-network-ipam', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'}, 'oper': 'CREATE', 'request-id': 'req-df205b68-67d9-4f2c-b580-8ff5778bfd21', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:56:57.106 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-fabric'], 'obj_dict': {'disable_vlan_vn_uniqueness_check': False, 'fabric_enterprise_style': True, 'fq_name': ['default-global-system-config', 'default-fabric'], 'id_perms': {'created': '2025-09-01T01:56:55.201463', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.201463', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11546110393382400586, 'uuid_mslong': 564015743764021666}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '07d3c954-a6c8-41a2-a03c-063a9700f24a'}, 'oper': 'CREATE', 'request-id': 'req-4813a06e-0df6-48ac-8e2c-8b88614bbfa4', 'type': 'fabric', 'uuid': '07d3c954-a6c8-41a2-a03c-063a9700f24a'} 09/01/2025 01:56:57.107 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management'], 'obj_dict': {'fq_name': ['default-policy-management'], 'id_perms': {'created': '2025-09-01T01:56:55.249163', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.249163', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12496563589972585086, 'uuid_mslong': 12955305086349234001}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b3ca7d55-fca8-4f51-ad6c-b6601d656a7e'}, 'oper': 'CREATE', 'request-id': 'req-d87972f5-642f-46ab-b0c1-8eba075002e3', 'type': 'policy_management', 'uuid': 'b3ca7d55-fca8-4f51-ad6c-b6601d656a7e'} 09/01/2025 01:56:57.108 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-namespace'], 'obj_dict': {'_type': 'namespace', '_uuid': None, 'fq_name': ['default-domain', 'default-namespace'], 'id_perms': {'created': '2025-09-01T01:56:55.266480', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.266480', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9736686625381238238, 'uuid_mslong': 2525012586862363280}}, 'name': 'default-namespace', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '230aa591-d552-4290-871f-a8e60b2545de'}, 'oper': 'CREATE', 'request-id': 'req-4416d6c5-6871-46ee-be5e-cbb3135def6a', 'type': 'namespace', 'uuid': '230aa591-d552-4290-871f-a8e60b2545de'} 09/01/2025 01:56:57.109 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-policy-management', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:56:55.332205', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.332205', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11711318808577496212, 'uuid_mslong': 18343649169922408591}}, 'parent_type': 'policy-management', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'}, 'oper': 'CREATE', 'request-id': 'req-ecac6868-ccef-4c84-98ec-099cc44c7916', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:56:57.110 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-306b9c9f-d99c-44af-b492-a160c20804db', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:56:57.110 7f5eef38bea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.110 7f5eef38bea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:56:55.332205", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:56:57.110 7f5eef38bea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.115 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-9281c619-6790-4bc7-b805-d0139a0af1c2', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:56:57.115 7f5eef38bea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.115 7f5eef38bea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:56:55.369695", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:56:57.116 7f5eef38bea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.122 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:56:55.457028', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.457028', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12936707290311336679, 'uuid_mslong': 11906273895913767761}}, 'igmp_enable': False, 'is_provider_network': True, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 2}, 'oper': 'CREATE', 'request-id': 'req-37613a31-4404-490d-92b4-61028185c78a', 'type': 'virtual_network', 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'} 09/01/2025 01:56:57.123 7f5eef38bea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.123 7f5eef38bea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:56:55.369695", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:56:57.123 7f5eef38bea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.123 7f5eef38bea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.123 7f5eef38bea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:ip-fabric elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "ip-fabric"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11906273895913767761, "uuid_lslong": 12936707290311336679}, "enable": true, "created": "2025-09-01T01:56:55.457028", "last_modified": "2025-09-01T01:56:55.457028", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : truelayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "a53b9506-b472-4351-b388-6abd2a36a2e7"virtual_network_category : "tenant"virtual_network_network_id : 2 } deleted = False >> 09/01/2025 01:56:57.123 7f5eef38bea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.126 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:56:55.524567', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.524567', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10705226111173909447, 'uuid_mslong': 18021289239621946886}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'fa187b15-223c-4e06-9490-9a6decc1dbc7'}, 'oper': 'CREATE', 'request-id': 'req-f8051ea6-43f3-4caa-a195-aa31a0cad417', 'type': 'routing_instance', 'uuid': 'fa187b15-223c-4e06-9490-9a6decc1dbc7'} 09/01/2025 01:56:57.127 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'id_perms': {'created': '2025-09-01T01:56:55.607419', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.607419', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13237806683607020471, 'uuid_mslong': 2656251315208078103}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7'}, 'oper': 'CREATE', 'request-id': 'req-a74879c8-3304-4f82-a247-d8bb663777ad', 'type': 'routing_instance', 'uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7'} 09/01/2025 01:56:57.128 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', '__link_local__'], 'id_perms': {'created': '2025-09-01T01:56:55.815663', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.815663', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9648904630498389471, 'uuid_mslong': 3583347645795025512}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': '31ba9bb2-ce58-4268-85e7-cba3ac7a6ddf', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 3}, 'oper': 'CREATE', 'request-id': 'req-28a37ca1-a648-4037-9f53-be0ec69df7e9', 'type': 'virtual_network', 'uuid': '31ba9bb2-ce58-4268-85e7-cba3ac7a6ddf'} 09/01/2025 01:56:57.128 7f5eef38bea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.128 7f5eef38bea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:__link_local__ elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "__link_local__"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3583347645795025512, "uuid_lslong": 9648904630498389471}, "enable": true, "created": "2025-09-01T01:56:55.815663", "last_modified": "2025-09-01T01:56:55.815663", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "31ba9bb2-ce58-4268-85e7-cba3ac7a6ddf"virtual_network_category : "tenant"virtual_network_network_id : 3 } deleted = False >> 09/01/2025 01:56:57.128 7f5eef38bea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.131 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'id_perms': {'created': '2025-09-01T01:56:55.915661', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.915661', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12680421812887378013, 'uuid_mslong': 1807274221672940190}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '1914ba54-4a89-429e-aff9-e8747f44585d'}, 'oper': 'CREATE', 'request-id': 'req-ad500610-9c33-4918-a5fb-a1f8bf2eb587', 'type': 'routing_instance', 'uuid': '1914ba54-4a89-429e-aff9-e8747f44585d'} 09/01/2025 01:56:57.132 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'dci-network'], 'id_perms': {'created': '2025-09-01T01:56:55.995390', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:55.995390', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11584685736362511906, 'uuid_mslong': 10402109786093929934}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': '905bb7bc-dabe-45ce-a0c5-124c866c0e22', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 4}, 'oper': 'CREATE', 'request-id': 'req-c2a75a81-5d3e-4b5d-9748-c4d2b21e495a', 'type': 'virtual_network', 'uuid': '905bb7bc-dabe-45ce-a0c5-124c866c0e22'} 09/01/2025 01:56:57.133 7f5eeffb1360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.133 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:dci-network elements = { fabric_snat : falseflood_unknown_unicast : falsefq_name : ["default-domain", "default-project", "dci-network"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10402109786093929934, "uuid_lslong": 11584685736362511906}, "enable": true, "created": "2025-09-01T01:56:55.995390", "last_modified": "2025-09-01T01:56:55.995390", "description": null, "user_visible": true, "creator": null}igmp_enable : falseis_provider_network : falselayer2_control_word : falsemac_aging_time : 300mac_ip_learning_enable : falsemac_learning_enabled : falseparent_type : "project"pbb_etree_enable : falsepbb_evpn_enable : falseperms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}port_security_enabled : trueuuid : "905bb7bc-dabe-45ce-a0c5-124c866c0e22"virtual_network_category : "tenant"virtual_network_network_id : 4 } deleted = False >> 09/01/2025 01:56:57.133 7f5eeffb1360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-3..dummy'] 09/01/2025 01:56:57.137 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'id_perms': {'created': '2025-09-01T01:56:56.046788', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.046788', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10920313239998599314, 'uuid_mslong': 12971198674826251470}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'b402f478-6c09-4cce-978c-bf08921c9092'}, 'oper': 'CREATE', 'request-id': 'req-a8ba60cd-b916-43a8-80e7-17afa153db2c', 'type': 'routing_instance', 'uuid': 'b402f478-6c09-4cce-978c-bf08921c9092'} 09/01/2025 01:56:57.138 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'id_perms': {'created': '2025-09-01T01:56:56.085853', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.085853', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9698331334556144440, 'uuid_mslong': 2776894989461965850}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': '26898347-c171-4c1a-8697-64f6fe425338'}, 'oper': 'CREATE', 'request-id': 'req-2c238a28-889a-4823-b8f5-804a7f5d366a', 'type': 'routing_instance', 'uuid': '26898347-c171-4c1a-8697-64f6fe425338'} 09/01/2025 01:56:57.139 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'UveConfigReq.err_info.*.', 'operand2': {'json_value': 'True'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'id_perms': {'created': '2025-09-01T01:56:56.131127', 'description': 'Bottle request size limit exceeded.', 'last_modified': '2025-09-01T01:56:56.131127', 'uuid': {'uuid_lslong': 13504318197043628231, 'uuid_mslong': 13387747455374148977}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b9cad561-1c37-4571-bb68-f9d9eca108c7', 'uve_keys': {'uve_key': ['config-node']}}, 'oper': 'CREATE', 'request-id': 'req-e8f1449e-c5bd-4121-907a-ab7a4cf91046', 'type': 'alarm', 'uuid': 'b9cad561-1c37-4571-bb68-f9d9eca108c7'} 09/01/2025 01:56:57.140 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'id_perms': {'created': '2025-09-01T01:56:56.174377', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.174377', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13073416659995647186, 'uuid_mslong': 12723500195717335128}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'b092f404-1958-4858-b56e-1b2fb3b6dcd2'}, 'oper': 'CREATE', 'request-id': 'req-0bae8712-b214-418d-ac76-e311c0fd5d87', 'type': 'routing_instance', 'uuid': 'b092f404-1958-4858-b56e-1b2fb3b6dcd2'} 09/01/2025 01:56:57.141 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'obj_dict': {'configured_security_group_id': 0, 'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'id_perms': {'created': '2025-09-01T01:56:56.254374', 'creator': None, 'description': 'Security group with no rules', 'enable': True, 'last_modified': '2025-09-01T01:56:56.254374', 'permissions': None, 'user_visible': True, 'uuid': {'uuid_lslong': 9252451694406411525, 'uuid_mslong': 7118527704291427150}}, 'parent_type': 'project', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': []}, 'security_group_id': 8000001, 'uuid': '62ca173b-4211-434e-8067-4fcb628e2505'}, 'oper': 'CREATE', 'request-id': 'req-1e8ce902-f857-4bc8-af81-dd063ae50c11', 'type': 'security_group', 'uuid': '62ca173b-4211-434e-8067-4fcb628e2505'} 09/01/2025 01:56:57.143 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-discovery-service-assignment'], 'obj_dict': {'fq_name': ['default-discovery-service-assignment'], 'id_perms': {'created': '2025-09-01T01:56:56.300890', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.300890', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12095067026772171269, 'uuid_mslong': 2018391152408088199}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1c02c40f-2913-4e87-a7da-4f60e14b9a05'}, 'oper': 'CREATE', 'request-id': 'req-544fb616-9375-414c-bcb3-b18bad8084e2', 'type': 'discovery_service_assignment', 'uuid': '1c02c40f-2913-4e87-a7da-4f60e14b9a05'} 09/01/2025 01:56:57.144 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'obj_dict': {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'id_perms': {'created': '2025-09-01T01:56:56.342807', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.342807', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11842624611448522391, 'uuid_mslong': 15265082910612213670}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd3d87807-81ed-4fa6-a459-7456d5b00697'}, 'oper': 'CREATE', 'request-id': 'req-cc4c10bc-5f7f-4f3a-a9a1-968ea28463c3', 'type': 'global_qos_config', 'uuid': 'd3d87807-81ed-4fa6-a459-7456d5b00697'} 09/01/2025 01:56:57.145 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'id_perms': {'created': '2025-09-01T01:56:56.383868', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.383868', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13493431935977500889, 'uuid_mslong': 18362981106062149845}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 8}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}, {'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '::ffff', 'ip_prefix_len': 104}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fed66a06-540b-48d5-bb42-4cdac0d9b0d9'}, 'oper': 'CREATE', 'request-id': 'req-70f11221-14e6-4403-8cc9-ed5590435d47', 'type': 'network_ipam', 'uuid': 'fed66a06-540b-48d5-bb42-4cdac0d9b0d9'} 09/01/2025 01:56:57.153 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['label'], 'obj_dict': {'display_name': 'label', 'fq_name': ['label'], 'id_perms': {'created': '2025-09-01T01:56:56.422823', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.422823', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12244024477607660426, 'uuid_mslong': 5547407232025379499}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0000', 'uuid': '4cfc597c-714e-42ab-a9eb-836408fe0b8a'}, 'oper': 'CREATE', 'request-id': 'req-d4637f2a-5f21-480d-8d6e-35e07207c87a', 'type': 'tag_type', 'uuid': '4cfc597c-714e-42ab-a9eb-836408fe0b8a'} 09/01/2025 01:56:57.156 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['application'], 'obj_dict': {'display_name': 'application', 'fq_name': ['application'], 'id_perms': {'created': '2025-09-01T01:56:56.453512', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.453512', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11358561686192514322, 'uuid_mslong': 3932761210100401007}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0001', 'uuid': '3693f97b-e278-436f-9da1-b7ac60dde512'}, 'oper': 'CREATE', 'request-id': 'req-42103f20-e31d-4771-8031-8a744709142b', 'type': 'tag_type', 'uuid': '3693f97b-e278-436f-9da1-b7ac60dde512'} 09/01/2025 01:56:57.158 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['tier'], 'obj_dict': {'display_name': 'tier', 'fq_name': ['tier'], 'id_perms': {'created': '2025-09-01T01:56:56.495204', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.495204', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10040837297144479750, 'uuid_mslong': 12939870066474175391}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0002', 'uuid': 'b393a744-5fe7-4b9f-8b58-3852b7ca4006'}, 'oper': 'CREATE', 'request-id': 'req-2eb082a5-14cf-4651-88d3-20fd14afcf78', 'type': 'tag_type', 'uuid': 'b393a744-5fe7-4b9f-8b58-3852b7ca4006'} 09/01/2025 01:56:57.161 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['deployment'], 'obj_dict': {'display_name': 'deployment', 'fq_name': ['deployment'], 'id_perms': {'created': '2025-09-01T01:56:56.519889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.519889', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12576522111879090180, 'uuid_mslong': 8119457381763793107}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0003', 'uuid': '70ae1b78-50f4-40d3-ae88-c83a3c661004'}, 'oper': 'CREATE', 'request-id': 'req-a1ce3229-800e-4646-b543-322eca568ebf', 'type': 'tag_type', 'uuid': '70ae1b78-50f4-40d3-ae88-c83a3c661004'} 09/01/2025 01:56:57.163 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['site'], 'obj_dict': {'display_name': 'site', 'fq_name': ['site'], 'id_perms': {'created': '2025-09-01T01:56:56.539546', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.539546', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12138098882871787574, 'uuid_mslong': 9632017415455261418}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0004', 'uuid': '85abccce-8696-42ea-a873-309fa2238036'}, 'oper': 'CREATE', 'request-id': 'req-cd9e7c89-09ed-4334-9b45-42b3501ff7d1', 'type': 'tag_type', 'uuid': '85abccce-8696-42ea-a873-309fa2238036'} 09/01/2025 01:56:57.171 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_fwaas'], 'obj_dict': {'display_name': 'neutron_fwaas', 'fq_name': ['neutron_fwaas'], 'id_perms': {'created': '2025-09-01T01:56:56.573898', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.573898', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13603708349514909711, 'uuid_mslong': 510975407527969929}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0005', 'uuid': '0717596d-06aa-4489-bcca-14ab18da080f'}, 'oper': 'CREATE', 'request-id': 'req-56643726-f616-47cb-be60-597ac98b1478', 'type': 'tag_type', 'uuid': '0717596d-06aa-4489-bcca-14ab18da080f'} 09/01/2025 01:56:57.176 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_tag'], 'obj_dict': {'display_name': 'neutron_tag', 'fq_name': ['neutron_tag'], 'id_perms': {'created': '2025-09-01T01:56:56.606455', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.606455', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13819367904009731180, 'uuid_mslong': 1255328771868151227}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0006', 'uuid': '116bd2dd-f387-4dbb-bfc8-41e3faef3c6c'}, 'oper': 'CREATE', 'request-id': 'req-04d15dfe-fe35-4744-8e65-fd26c607d98b', 'type': 'tag_type', 'uuid': '116bd2dd-f387-4dbb-bfc8-41e3faef3c6c'} 09/01/2025 01:56:57.180 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-f4232883-035d-44ed-a6a2-01ffd63aadf8', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:56:57.186 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-api-access-list'], 'obj_dict': {'api_access_list_entries': {'rbac_rule': [{'rule_field': '', 'rule_object': 'fqname-to-id', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'id-to-fqname', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'useragent-kv', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'documentation', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': '/', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'watch', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': 'logical_router_gateway_external', 'rule_object': 'logical-router', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}]}, 'fq_name': ['default-global-system-config', 'default-api-access-list'], 'id_perms': {'created': '2025-09-01T01:56:56.757987', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:56.757987', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12348196333925518916, 'uuid_mslong': 7523197845815445895}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6867c494-c6d0-4987-ab5d-9b2464694e44'}, 'oper': 'CREATE', 'request-id': 'req-2bbabfe3-2350-42c5-831e-308deb4fd935', 'type': 'api_access_list', 'uuid': '6867c494-c6d0-4987-ab5d-9b2464694e44'} 09/01/2025 01:56:57.186 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:56:57.037005', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:56:57.037005', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11816159058979396378, 'uuid_mslong': 7782728632934810873}}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6c01ce74-fd57-44f9-a3fb-6e0f37fb1b1a'}, 'oper': 'CREATE', 'request-id': 'req-4286a9d4-4d79-43fe-9050-fb0d4d33a1a5', 'type': 'application_policy_set', 'uuid': '6c01ce74-fd57-44f9-a3fb-6e0f37fb1b1a'} 09/01/2025 01:56:57.187 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:56:57.187 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-b756271d-0a67-423d-97ba-34d94a540ca1', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:56:57.188 7f5eeffb1360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.188 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project:default-application-policy-set elements = { all_applications : truefq_name : ["default-domain", "default-project", "default-application-policy-set"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7782728632934810873, "uuid_lslong": 11816159058979396378}, "enable": true, "created": "2025-09-01T01:56:57.037005", "last_modified": "2025-09-01T01:56:57.037005", "description": null, "user_visible": true, "creator": null}parent_type : "project"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6c01ce74-fd57-44f9-a3fb-6e0f37fb1b1a" } deleted = False >> 09/01/2025 01:56:57.188 7f5eeffb1360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.192 7f5eeffb1360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:57.192 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:56:57.088017", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:56:57.192 7f5eeffb1360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:58.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:58.161 7f5eef329a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:58.162 7f5eef329a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:58.162 7f5eef329a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:58.162 7f5eef329a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:58.162 7f5eef329a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:56:58.163 7f5eef329a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:59.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:56:59.890 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:59.891 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:59.892 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:59.893 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:59.926 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:59.927 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:59.928 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:56:59.945 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:56:59.946 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:56:59.947 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:56:59.948 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:00.000 7f5eef329180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'] not found in DB: Unknown id: config_database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. 09/01/2025 01:57:00.009 7f5eef329180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. url = http://10.0.0.131:8082/config-database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_config_database_node.py remote_ip = 10.0.0.38 body = {"config-database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."], "parent_type": "global-system-config", "config_database_node_ip_address": "10.0.0.38", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:00.027 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'] not found in DB: Unknown id: config_database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. 09/01/2025 01:57:00.035 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. url = http://10.0.0.131:8082/config-database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_config_database_node.py remote_ip = 10.0.0.131 body = {"config-database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."], "parent_type": "global-system-config", "config_database_node_ip_address": "10.0.0.131", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:00.039 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:57:00.032197', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:00.032197', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13388812389776392619, 'uuid_mslong': 15785457609365275379}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'db11360f-8f34-42f3-b9ce-9dee78f881ab'}, 'oper': 'CREATE', 'request-id': 'req-c9f3a7e2-91ed-48bb-9bb4-430350284d47', 'type': 'config_database_node', 'uuid': 'db11360f-8f34-42f3-b9ce-9dee78f881ab'} 09/01/2025 01:57:00.039 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:57:00.040 7f5eef2dd0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.040 7f5eef2dd0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { config_database_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15785457609365275379, "uuid_lslong": 13388812389776392619}, "enable": true, "created": "2025-09-01T01:57:00.032197", "last_modified": "2025-09-01T01:57:00.032197", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "db11360f-8f34-42f3-b9ce-9dee78f881ab" } deleted = False >> 09/01/2025 01:57:00.040 7f5eef2dd0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.056 7f5eef329360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'] not found in DB: Unknown id: config_database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. 09/01/2025 01:57:00.065 7f5eef329360 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. url = http://10.0.0.131:8082/config-database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_config_database_node.py remote_ip = 10.0.0.249 body = {"config-database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."], "parent_type": "global-system-config", "config_database_node_ip_address": "10.0.0.249", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:00.070 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:57:00.063354', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:00.063354', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11110068176641177895, 'uuid_mslong': 12066910213316166934}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a77646e7-e13e-4516-9a2e-e4249b11b927'}, 'oper': 'CREATE', 'request-id': 'req-a43d83da-0fa1-4111-95a4-17bb16e59e2d', 'type': 'config_database_node', 'uuid': 'a77646e7-e13e-4516-9a2e-e4249b11b927'} 09/01/2025 01:57:00.070 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:57:00.071 7f5eef2dd0e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.071 7f5eef2dd0e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { config_database_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12066910213316166934, "uuid_lslong": 11110068176641177895}, "enable": true, "created": "2025-09-01T01:57:00.063354", "last_modified": "2025-09-01T01:57:00.063354", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a77646e7-e13e-4516-9a2e-e4249b11b927" } deleted = False >> 09/01/2025 01:57:00.071 7f5eef2dd0e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.123 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:57:00.115323', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:00.115323', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10321475497656808872, 'uuid_mslong': 3511137345545324278}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '30ba10d0-1ed6-46f6-8f3d-3f4793bb75a8'}, 'oper': 'CREATE', 'request-id': 'req-b40c2f60-6ce9-4db0-bce8-7f38d3144ab5', 'type': 'config_database_node', 'uuid': '30ba10d0-1ed6-46f6-8f3d-3f4793bb75a8'} 09/01/2025 01:57:00.123 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:57:00.123 7f5eef329180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.124 7f5eef329180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { config_database_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3511137345545324278, "uuid_lslong": 10321475497656808872}, "enable": true, "created": "2025-09-01T01:57:00.115323", "last_modified": "2025-09-01T01:57:00.115323", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "30ba10d0-1ed6-46f6-8f3d-3f4793bb75a8" } deleted = False >> 09/01/2025 01:57:00.124 7f5eef329180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:00.249 7f5eef38be00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.249 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:00.249 7f5eef38be00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:01.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:02.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:02.163 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:02.164 7f5eef3294a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:02.164 7f5eef3294a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:02.164 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:02.164 7f5eef3294a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:02.164 7f5eef3294a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:02.164 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:02.165 7f5eef3294a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:02.277 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:02.341 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:02.342 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:02.343 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:02.344 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:02.400 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'opencontrail'] not found in DB: Unknown id: service_appliance_set default-global-system-config:opencontrail 09/01/2025 01:57:02.411 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:opencontrail url = http://10.0.0.131:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "opencontrail"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "opencontrail"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:02.448 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'opencontrail'], 'obj_dict': {'display_name': 'opencontrail', 'fq_name': ['default-global-system-config', 'opencontrail'], 'id_perms': {'created': '2025-09-01T01:57:02.440530', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:02.440530', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9947658057775578254, 'uuid_mslong': 15018853803976770201}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver', 'uuid': 'd06daff6-1f8f-4299-8a0d-2e4c70c3fc8e'}, 'oper': 'CREATE', 'request-id': 'req-52500de4-17a7-4de2-b15c-8b08070421bd', 'type': 'service_appliance_set', 'uuid': 'd06daff6-1f8f-4299-8a0d-2e4c70c3fc8e'} 09/01/2025 01:57:02.627 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:02.627 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:02.627 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:02.627 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:02.628 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:02.629 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:02.629 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:02.629 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:02.629 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:02.629 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:02.630 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:02.630 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:02.630 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:02.631 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:02.632 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:02.632 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:02.633 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:02.633 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:02.633 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:02.691 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'native'] not found in DB: Unknown id: service_appliance_set default-global-system-config:native 09/01/2025 01:57:02.692 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:native url = http://10.0.0.131:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "native"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "native"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:02.750 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'native'], 'obj_dict': {'display_name': 'native', 'fq_name': ['default-global-system-config', 'native'], 'id_perms': {'created': '2025-09-01T01:57:02.726776', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:02.726776', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12665353147798690348, 'uuid_mslong': 15114292327656474848}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver', 'uuid': 'd1c0c0cb-0560-48e0-afc4-5f95816a9a2c'}, 'oper': 'CREATE', 'request-id': 'req-ab8e49ae-2f2a-44dd-b9a9-eb0a26c2d5ca', 'type': 'service_appliance_set', 'uuid': 'd1c0c0cb-0560-48e0-afc4-5f95816a9a2c'} 09/01/2025 01:57:02.911 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 09/01/2025 01:57:02.911 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 09/01/2025 01:57:02.911 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 09/01/2025 01:57:02.911 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 09/01/2025 01:57:02.911 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 09/01/2025 01:57:03.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:03.429 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'analyzer-template'] not found in DB: Unknown id: service_template default-domain:analyzer-template 09/01/2025 01:57:03.430 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:analyzer-template url = http://10.0.0.131:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "analyzer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": null, "service_type": "analyzer", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "analyzer-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:03.497 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'analyzer-template'], 'obj_dict': {'display_name': 'analyzer-template', 'fq_name': ['default-domain', 'analyzer-template'], 'id_perms': {'created': '2025-09-01T01:57:03.481825', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:03.481825', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12235799898289269312, 'uuid_mslong': 2310985238325185939}}, 'parent_type': 'domain', 'parent_uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': [{'tenant': 'domain:4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': None, 'service_scaling': False, 'service_type': 'analyzer', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '201244d4-838b-4d93-a9ce-4b2df04aea40'}, 'oper': 'CREATE', 'request-id': 'req-d980c06c-934c-4626-9142-54a8bc4e6296', 'type': 'service_template', 'uuid': '201244d4-838b-4d93-a9ce-4b2df04aea40'} 09/01/2025 01:57:03.546 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'nat-template'] not found in DB: Unknown id: service_template default-domain:nat-template 09/01/2025 01:57:03.547 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:nat-template url = http://10.0.0.131:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "nat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "firewall", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "nat-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:03.704 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'nat-template'], 'obj_dict': {'display_name': 'nat-template', 'fq_name': ['default-domain', 'nat-template'], 'id_perms': {'created': '2025-09-01T01:57:03.669132', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:03.669132', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10175594740631010065, 'uuid_mslong': 13519165387915215512}}, 'parent_type': 'domain', 'parent_uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': [{'tenant': 'domain:4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': 'bb9db94a-ee71-4298-8d36-f98105a4fb11'}, 'oper': 'CREATE', 'request-id': 'req-689133ca-ff72-4bdc-99c3-5d1d48a22dcf', 'type': 'service_template', 'uuid': 'bb9db94a-ee71-4298-8d36-f98105a4fb11'} 09/01/2025 01:57:03.774 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'netns-snat-template'] not found in DB: Unknown id: service_template default-domain:netns-snat-template 09/01/2025 01:57:03.775 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:netns-snat-template url = http://10.0.0.131:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "netns-snat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "source-nat", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "netns-snat-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:03.868 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'netns-snat-template'], 'obj_dict': {'display_name': 'netns-snat-template', 'fq_name': ['default-domain', 'netns-snat-template'], 'id_perms': {'created': '2025-09-01T01:57:03.842557', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:03.842557', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11737752235013800087, 'uuid_mslong': 17996682902188607714}}, 'parent_type': 'domain', 'parent_uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': [{'tenant': 'domain:4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'source-nat', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': 'f9c10fbf-c04c-48e2-a2e4-df77ed8b8097'}, 'oper': 'CREATE', 'request-id': 'req-11a89bca-f426-4dfd-87b7-79fee9b6f4c4', 'type': 'service_template', 'uuid': 'f9c10fbf-c04c-48e2-a2e4-df77ed8b8097'} 09/01/2025 01:57:03.933 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'haproxy-loadbalancer-template'] not found in DB: Unknown id: service_template default-domain:haproxy-loadbalancer-template 09/01/2025 01:57:03.934 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:haproxy-loadbalancer-template url = http://10.0.0.131:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "haproxy-loadbalancer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "loadbalancer", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "haproxy-loadbalancer-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:04.000 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'obj_dict': {'display_name': 'haproxy-loadbalancer-template', 'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'id_perms': {'created': '2025-09-01T01:57:03.988005', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:03.988005', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13175936714501905230, 'uuid_mslong': 7566801988425173734}}, 'parent_type': 'domain', 'parent_uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': [{'tenant': 'domain:4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'loadbalancer', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '6902ae51-5dcd-4ae6-b6da-54a1fae9db4e'}, 'oper': 'CREATE', 'request-id': 'req-02deb93d-74aa-48c2-a145-228a381c47ae', 'type': 'service_template', 'uuid': '6902ae51-5dcd-4ae6-b6da-54a1fae9db4e'} 09/01/2025 01:57:04.098 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'docker-template'] not found in DB: Unknown id: service_template default-domain:docker-template 09/01/2025 01:57:04.099 7f5eef38be00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:docker-template url = http://10.0.0.131:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.38 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "docker-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "transparent", "service_type": "firewall", "image_name": "ubuntu", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "vrouter-instance", "availability_zone_enable": false, "vrouter_instance_type": "docker", "instance_data": "{\"command\":\"/bin/bash\"}"}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "docker-template"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:04.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:04.172 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'docker-template'], 'obj_dict': {'display_name': 'docker-template', 'fq_name': ['default-domain', 'docker-template'], 'id_perms': {'created': '2025-09-01T01:57:04.158033', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:04.158033', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13002801556611157938, 'uuid_mslong': 4996765549094718435}}, 'parent_type': 'domain', 'parent_uuid': '4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': [{'tenant': 'domain:4bb7dde1-472f-41f0-9da0-5eb4249cd377', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': 'ubuntu', 'instance_data': '{"command":"/bin/bash"}', 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'transparent', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'vrouter-instance', 'version': 1, 'vrouter_instance_type': 'docker'}, 'uuid': '455813ca-fbb3-4be3-b473-3b1fbab44fb2'}, 'oper': 'CREATE', 'request-id': 'req-2a319788-9600-4888-83e4-231780422e27', 'type': 'service_template', 'uuid': '455813ca-fbb3-4be3-b473-3b1fbab44fb2'} 09/01/2025 01:57:05.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:06.104 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-a6d570db-61ed-483e-9d29-23c0445f8de7', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:57:06.123 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.123 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:57:06.086429", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:57:06.123 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:06.164 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:06.165 7f5ef5536e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:06.165 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:06.165 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.166 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.167 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:06.167 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.178 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:06.178 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:06.179 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:06.180 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:06.181 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:06.181 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:06.181 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:06.181 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:06.181 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:06.275 7f5eef38bc20 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'] not found in DB: Unknown id: config_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. 09/01/2025 01:57:06.280 7f5eef38bc20 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. url = http://10.0.0.131:8082/config-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_config_node.py remote_ip = 10.0.0.131 body = {"config-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."], "parent_type": "global-system-config", "config_node_ip_address": "10.0.0.131", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:06.317 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-4817b85a-0381-44d9-98b1-b4cc92223331', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:57:06.327 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'config_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:57:06.307811', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:06.307811', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13128683577167256048, 'uuid_mslong': 10033920323497510113}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8b3fa55f-4aa4-44e1-b632-7427998f19f0'}, 'oper': 'CREATE', 'request-id': 'req-b2c1949a-b9e4-4de8-b1f2-23c0b93c352f', 'type': 'config_node', 'uuid': '8b3fa55f-4aa4-44e1-b632-7427998f19f0'} 09/01/2025 01:57:06.327 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:06.328 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.328 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { config_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10033920323497510113, "uuid_lslong": 13128683577167256048}, "enable": true, "created": "2025-09-01T01:57:06.307811", "last_modified": "2025-09-01T01:57:06.307811", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "8b3fa55f-4aa4-44e1-b632-7427998f19f0" } deleted = False >> 09/01/2025 01:57:06.328 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.496 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-178dcc85-7d0f-4b7f-a81d-367ea0808919', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:57:06.500 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.500 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:57:06.330561", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:57:06.500 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:07.346 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-627a4ee1-7b6b-4f96-97ad-6e4314a60029', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:57:07.351 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.351 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:57:07.333259", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:57:07.351 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.535 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-d73327dd-fcdc-48c2-9ec5-f2dafab03c46', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:57:07.744 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-34de5f60-0dc2-44ba-9d39-a1c3bc2a7184', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:57:07.782 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.783 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:57:07.735619", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:57:07.783 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:08.450 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:08.450 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:08.451 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:08.452 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:08.453 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:08.453 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:08.453 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:08.453 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:08.453 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:08.555 7f5eef329f40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'] not found in DB: Unknown id: config_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. 09/01/2025 01:57:08.561 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. url = http://10.0.0.131:8082/config-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_config_node.py remote_ip = 10.0.0.38 body = {"config-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."], "parent_type": "global-system-config", "config_node_ip_address": "10.0.0.38", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:08.597 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'config_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:57:08.589359', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:08.589359', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9854752206779726768, 'uuid_mslong': 3885232349240774114}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0'}, 'oper': 'CREATE', 'request-id': 'req-080b3d52-9fcf-4ddd-853b-6402086b7f46', 'type': 'config_node', 'uuid': '35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0'} 09/01/2025 01:57:08.597 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:08.598 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.598 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { config_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3885232349240774114, "uuid_lslong": 9854752206779726768}, "enable": true, "created": "2025-09-01T01:57:08.589359", "last_modified": "2025-09-01T01:57:08.589359", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0" } deleted = False >> 09/01/2025 01:57:08.598 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:10.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:10.166 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:10.167 7f5ef5536e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:10.167 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:10.167 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:10.167 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:10.167 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:10.167 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:10.168 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:11.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:12.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:13.153 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:14.155 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:14.167 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:14.169 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:14.170 7f5ef4bc0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:14.170 7f5ef4bc0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:14.170 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:14.170 7f5ef4bc0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:14.170 7f5ef4bc0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.157 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:16.157 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:17.158 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:18.160 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:18.169 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:18.170 7f5ef4bc0360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:18.170 7f5ef4bc0360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:18.170 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:18.170 7f5ef4bc0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:18.171 7f5ef4bc0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.306 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:18.306 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:18.306 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:18.307 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:18.308 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:18.309 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:18.309 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:18.309 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:18.309 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:18.309 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:18.413 7f5ef4caa4a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'] not found in DB: Unknown id: config_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. 09/01/2025 01:57:18.419 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. url = http://10.0.0.131:8082/config-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_config_node.py remote_ip = 10.0.0.249 body = {"config-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."], "parent_type": "global-system-config", "config_node_ip_address": "10.0.0.249", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:18.450 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'config_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:57:18.438111', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.438111', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11481679389149772070, 'uuid_mslong': 9952658429463971103}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8a1ef21c-32ff-451f-9f57-1e9267323926'}, 'oper': 'CREATE', 'request-id': 'req-29c41265-556e-4193-aa1b-64401b965cb7', 'type': 'config_node', 'uuid': '8a1ef21c-32ff-451f-9f57-1e9267323926'} 09/01/2025 01:57:18.450 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:18.451 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.451 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { config_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9952658429463971103, "uuid_lslong": 11481679389149772070}, "enable": true, "created": "2025-09-01T01:57:18.438111", "last_modified": "2025-09-01T01:57:18.438111", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "8a1ef21c-32ff-451f-9f57-1e9267323926" } deleted = False >> 09/01/2025 01:57:18.451 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.162 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:20.164 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:21.165 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.167 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.171 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:22.172 7f5ef4caa4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:22.172 7f5ef4caa4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:22.172 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.173 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:22.174 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.170 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:24.171 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:25.173 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.174 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.174 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:26.175 7f5ef4caa4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:26.176 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:26.177 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:26.177 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:28.080 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:29.082 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.084 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.176 7f5eef38bc20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:30.177 7f5eef38bc20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:30.178 7f5eef38bc20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:30.178 7f5eef38bc20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:30.178 7f5eef38bc20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:30.178 7f5eef38bc20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:30.178 7f5eef38bc20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:30.179 7f5eef38bc20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.085 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:32.087 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:33.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.090 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.178 7f5eef329540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:34.180 7f5eef329540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:34.180 7f5eef329540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:34.180 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:34.180 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:34.180 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:34.180 7f5eef329540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:34.181 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:36.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:37.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:38.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:38.180 7f5eef329540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.182 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:38.183 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.183 7f5eef329540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.183 7f5eef329540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:38.183 7f5eef329540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:40.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:41.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.182 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:42.183 7f5ef5536e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:42.183 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:42.183 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:42.183 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.184 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.185 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:42.185 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.784 7f5eeffb1360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 09/01/2025 01:57:42.784 7f5eeffb1360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 09/01/2025 01:57:42.785 7f5eeffb1360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 09/01/2025 01:57:42.785 7f5eeffb1360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 09/01/2025 01:57:42.785 7f5eeffb1360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 09/01/2025 01:57:43.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:43.470 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:43.460769', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.460769', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11925350455766867329, 'uuid_mslong': 15648137040972304299}}, 'parent_type': 'security-group', 'parent_uuid': '62ca173b-4211-434e-8067-4fcb628e2505', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'}, 'oper': 'CREATE', 'request-id': 'req-e9e6110b-15a8-45e1-b6ff-9bbe5f1606d8', 'type': 'access_control_list', 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'} 09/01/2025 01:57:43.519 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:43.512816', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.512816', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11154437056324030764, 'uuid_mslong': 14414019718996774673}}, 'parent_type': 'security-group', 'parent_uuid': '62ca173b-4211-434e-8067-4fcb628e2505', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'}, 'oper': 'CREATE', 'request-id': 'req-a4d2e73b-68f2-4c38-a76b-9509eebec0d9', 'type': 'access_control_list', 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'} 09/01/2025 01:57:43.619 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-09-01T01:57:43.610873', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.610873', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13092489608894395420, 'uuid_mslong': 18272932050593923086}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'fd967ee2-c880-400e-b5b1-ddeeaba4b41c'}, 'oper': 'CREATE', 'request-id': 'req-667bdd9c-a819-416b-a362-7c8709dec24e', 'type': 'route_target', 'uuid': 'fd967ee2-c880-400e-b5b1-ddeeaba4b41c'} 09/01/2025 01:57:43.627 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 26898347-c171-4c1a-8697-64f6fe425338 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:__default__ url = http://10.0.0.131:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "routing-instance", "uuid": "26898347-c171-4c1a-8697-64f6fe425338", "ref-type": "route-target", "ref-uuid": "fd967ee2-c880-400e-b5b1-ddeeaba4b41c", "ref-fq-name": ["target:64512:8000000"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.646 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-d38d9b73-6e23-420e-ad61-26940b0d8417', 'type': 'routing_instance', 'uuid': '26898347-c171-4c1a-8697-64f6fe425338'} 09/01/2025 01:57:43.689 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-09-01T01:57:43.671641', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.671641', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13570455496092117547, 'uuid_mslong': 12041940091083179522}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'a71d90b6-3f8c-4602-bc53-f15fd4d16a2b'}, 'oper': 'CREATE', 'request-id': 'req-848a89e9-1bb6-4095-a058-f8b3aa42453b', 'type': 'route_target', 'uuid': 'a71d90b6-3f8c-4602-bc53-f15fd4d16a2b'} 09/01/2025 01:57:43.712 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-7fd48cc7-6227-4e3e-9208-830d1b311a69', 'type': 'routing_instance', 'uuid': 'b402f478-6c09-4cce-978c-bf08921c9092'} 09/01/2025 01:57:43.731 7f5eeffb1360 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000002'] not found in DB: Unknown id: route_target target:64512:8000002 09/01/2025 01:57:43.732 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000002 url = http://10.0.0.131:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000002"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000002"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.747 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-09-01T01:57:43.738435', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.738435', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13504741517806881909, 'uuid_mslong': 10871219233836909678}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': '96de544b-843e-446e-bb6a-7adbf8594c75'}, 'oper': 'CREATE', 'request-id': 'req-eafc60a5-2b5d-4bf9-8cf2-e091b0ef8565', 'type': 'route_target', 'uuid': '96de544b-843e-446e-bb6a-7adbf8594c75'} 09/01/2025 01:57:43.771 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-dd79017c-e717-4d14-ba03-dacf3e729740', 'type': 'routing_instance', 'uuid': 'b092f404-1958-4858-b56e-1b2fb3b6dcd2'} 09/01/2025 01:57:43.814 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-09-01T01:57:43.808184', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.808184', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10211255544152584447, 'uuid_mslong': 14240605882422477187}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'c5a0cb99-a021-4183-8db5-aad0dbf3f0ff'}, 'oper': 'CREATE', 'request-id': 'req-e8279a5b-a4e6-4fee-aa01-864a568ee38c', 'type': 'route_target', 'uuid': 'c5a0cb99-a021-4183-8db5-aad0dbf3f0ff'} 09/01/2025 01:57:43.823 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = fa187b15-223c-4e06-9490-9a6decc1dbc7 object_type = routing_instance identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.131:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "routing-instance", "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7", "ref-type": "route-target", "ref-uuid": "c5a0cb99-a021-4183-8db5-aad0dbf3f0ff", "ref-fq-name": ["target:64512:8000003"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.839 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-94fd4bc4-4c44-489d-8a70-3b0e401279f1', 'type': 'routing_instance', 'uuid': 'fa187b15-223c-4e06-9490-9a6decc1dbc7'} 09/01/2025 01:57:43.933 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:57:43.914671', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.914671', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12434635487439495936, 'uuid_mslong': 17013906886330174589}}, 'parent_type': 'virtual-network', 'parent_uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'}, 'oper': 'CREATE', 'request-id': 'req-7ee2abe6-6195-4e08-b9e2-a1db250d1fde', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 01:57:43.938 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = d92959bb-3bc9-4fab-a57f-5b0e6a06dd81 object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = http://10.0.0.131:8082/access-control-list/d92959bb-3bc9-4fab-a57f-5b0e6a06dd81 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"access-control-list": {"uuid": "d92959bb-3bc9-4fab-a57f-5b0e6a06dd81", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": 3520976145078303111}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.954 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-dacc4d4b-9503-4813-aead-82a4bd87ba77', 'type': 'access_control_list', 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'} 09/01/2025 01:57:43.973 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-28a2300c-ad5b-4055-b6ef-fd6c1340a532', 'type': 'access_control_list', 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'} 09/01/2025 01:57:44.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:45.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.184 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:46.186 7f5ef5536e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:46.187 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:46.187 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:46.187 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:46.187 7f5ef5536e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:46.187 7f5ef5536e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:46.187 7f5ef5536e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:47.462 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-b9cee00c-1b45-404e-af54-eadf4c384e92', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:47.689 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-bf1780e4-3d48-4dbf-97d0-5077fe0b1810', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:47.833 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-f41da2eb-8d35-4dcf-a758-8cf2d89040a0', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:48.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:49.091 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.25', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.25', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T01:57:49.074180', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.074180', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9621979915837513096, 'uuid_mslong': 12350586173020262281}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'}, 'oper': 'CREATE', 'request-id': 'req-c0a3b8a3-b9c0-47b4-aa6d-0410935f0b7e', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:49.092 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.093 7f5ef4c1c540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.093 7f5ef4c1c540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.074180", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:49.093 7f5ef4c1c540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:49.365 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-bbc43391-f503-4e5f-ae26-aa04ed6e08c3', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:49.717 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-01T01:57:49.704740', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.704740', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13465058781904455200, 'uuid_mslong': 269198390335917880}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-7d26a5a0-84f7-42ec-800e-7cdc9b85f184', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.768 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.129', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.129', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'id_perms': {'created': '2025-09-01T01:57:49.750291', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.750291', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11857703135272596087, 'uuid_mslong': 7997815229011673419}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'}, 'oper': 'CREATE', 'request-id': 'req-8f2e7ba5-2d05-4723-9303-eb4f0aa1e337', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:49.769 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.770 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.770 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:49.750291", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:49.770 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.798 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.193', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.193', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T01:57:49.778205', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.778205', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10075118084394099490, 'uuid_mslong': 1863496716102945915}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'}, 'oper': 'CREATE', 'request-id': 'req-e5b882a7-327b-41ce-b42a-1a0c876931dc', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:49.799 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.800 7f5eef38bea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.800 7f5eef38bea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:49.778205", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:49.800 7f5eef38bea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.839 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-02b807bd-14fd-4f1e-b82a-c25261b53d43', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.843 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-01T01:57:49.827148', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.827148', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9281412384941033885, 'uuid_mslong': 18032248181005373295}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4d39c21d-ea49-4fbd-881e-a4e63a6130e5', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:49.852 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-eaedbf1a-4d99-4ee5-a020-a5df73fc9289', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.889 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'oper': 'UPDATE', 'request-id': 'req-3b0d910e-4eff-4ddc-9b7a-42f2a96d0b1a', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:49.913 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-3b0d910e-4eff-4ddc-9b7a-42f2a96d0b1a', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:49.913 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.914 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:49.862739", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:49.916 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.922 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.922 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.859846", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:49.922 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.957 7f5eeffb1360 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 19dc7863-db6c-4c7b-8bd2-02847a6ca322 object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-2 url = http://10.0.0.131:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "bgp-router", "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322", "ref-type": "bgp-router", "ref-uuid": null, "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "operation": "ADD", "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.013 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'oper': 'UPDATE', 'request-id': 'req-8e0e1c75-cdee-4b81-a97d-47313a4410a5', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:50.027 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-8e0e1c75-cdee-4b81-a97d-47313a4410a5', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:50.028 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.028 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:49.990399", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:50.028 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.044 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-01T01:57:49.990142', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.990142', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13643264217144023489, 'uuid_mslong': 14422323711782898249}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-d198a7f7-9028-417a-81b9-278fb7ce7f83', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.045 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.045 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.981032", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:50.045 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.075 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-64ef5a1c-639c-4c3e-b6bf-c4d98fa5bb18', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:50.082 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-6248bd5b-43c4-48ea-a023-a61144970ae3', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:50.098 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-948d4f9f-cccb-41f5-b615-4fdb2635b0d4', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:50.106 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'oper': 'UPDATE', 'request-id': 'req-6f660d61-332b-4f6d-adba-f26f42913c20', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:50.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:50.117 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-6f660d61-332b-4f6d-adba-f26f42913c20', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:50.118 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.118 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:50.091698", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:50.118 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.122 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.123 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:50.085318", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:50.123 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.133 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-45aae796-5805-4664-8f68-8e9977d1075b', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:50.167 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.146892', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.146892', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12719190076713101290, 'uuid_mslong': 9355486960643295514}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-ab3956ca-bc75-4fc4-90b9-5fb4002eab77', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.187 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:50.188 7f5ef4caa4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:50.189 7f5ef4caa4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:50.189 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:50.189 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:50.189 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:50.189 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:50.189 7f5ef4caa4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:50.190 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.190 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:50.190 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.190 7f5ef4caa4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.190 7f5ef4caa4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:50.190 7f5ef4caa4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.258 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-01T01:57:50.244269', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.244269', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12345975188693128422, 'uuid_mslong': 12789451347087803108}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3ebf1ad0-4d29-4eb6-b68c-83cb5e0c2c57', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.278 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-4644b680-219f-41ac-93f0-2af58d6290f6', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.281 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-a1db2026-671d-43f2-beab-63ca7fd085d9', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.361 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-01T01:57:50.343229', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.343229', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410398409968651569, 'uuid_mslong': 348915920711273427}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-b3df3838-b993-445d-b342-9bdc65b9c290', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.452 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b46a4069-d0ac-41da-baab-6487258bee1f', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.454 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0ea88e73-e57b-474e-b375-5282426f050e', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.463 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-01T01:57:50.437138', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.437138', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10407682553178176690, 'uuid_mslong': 1486187007752490893}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-8cd87a25-7ae6-44da-ac17-b27b16f998eb', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.553 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-01T01:57:50.537284', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.537284', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12650030438488176505, 'uuid_mslong': 7230954819262302855}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-aa98e6eb-63fe-4a1e-abf9-ce5a05ca5b01', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:50.629 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-a927b894-3265-4f46-a249-ce671ebe63da', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.630 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-939925e0-8ddf-41ea-8f96-f1b4fbe1e932', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.647 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-01T01:57:50.636982', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.636982', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11945006111573889988, 'uuid_mslong': 15521484146632968236}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-772d7b53-1bf5-4336-9bd0-cded7ae36250', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:50.725 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.713917', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.713917', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13316427422065746893, 'uuid_mslong': 7023261606246303666}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-17a28f51-9acd-4cc4-9846-b019f65f92ca', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:50.792 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-95b9b3ba-1a0e-4ea3-9245-25030d1a7b8c', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.796 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-ff7124e2-b2c7-4010-879e-a96f7b647965', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.810 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-01T01:57:50.802208', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.802208', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10394145945911723784, 'uuid_mslong': 9299310539720442798}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4f2c310d-2af7-42f0-b0cc-347196801623', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:50.895 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.885227', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:50.885227', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11080958518291472377, 'uuid_mslong': 3112223694598000309}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-c2925086-dcce-45f5-ab84-4caad16fa4c3', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:50.968 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-6d2502b7-dea2-4907-81a1-e4b8a763d3ee', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.974 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-0b329c1d-2db3-448f-97e9-07c801904437', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.991 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.981229', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:50.981229', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10109986246743945171, 'uuid_mslong': 10619123826018765353}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-089282b9-7dc7-43b2-a6ef-7aaed496978e', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:51.074 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-01T01:57:51.062109', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.062109', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11650388398251038770, 'uuid_mslong': 10237949505899020380}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-ddf9ec54-1868-42ef-b16e-0e94d2b51aad', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:51.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:51.136 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-40c61117-53d3-4682-b6aa-ebab239dc2f6', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:51.140 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-372e3cc2-6865-4786-b23f-a0ea8df70b4e', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:51.157 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-01T01:57:51.143961', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.143961', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11278860563117070971, 'uuid_mslong': 11803345335316596501}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-1a9eb67e-f428-4a37-96a9-72466ae74d61', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:51.238 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:51.226178', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.226178', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11143410889604691103, 'uuid_mslong': 14282259945798913346}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-32967334-05f5-451f-82ec-3c34074acc51', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:51.301 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-b459fe8b-724f-444f-8092-ca948a7c4bb7', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:51.317 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-c1fbc6a9-f5aa-4b42-83a9-874da7552f7a', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:51.332 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-01T01:57:51.312606', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-01T01:57:51.312606', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13487801257560256231, 'uuid_mslong': 10235680329170243091}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-3c23439e-0ccb-4d2b-b98a-24cb37aaab0c', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:51.409 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-01T01:57:51.398898', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.398898', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10710412090359268505, 'uuid_mslong': 3024502048848363658}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-da6c1f03-ff69-4dc6-a1ce-b0a85f4b848d', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:51.476 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-86a74659-a45f-42ec-840b-7a8d9d5f0e2f', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:51.485 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-ff070e25-e45d-4fe3-be1d-003d05e1b568', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:51.503 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-01T01:57:51.495240', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.495240', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12799166691106454208, 'uuid_mslong': 15027170893856260812}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-94e1d593-b6f7-40fa-a257-80aad936d295', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:51.587 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-01T01:57:51.577756', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.577756', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10722918137435541080, 'uuid_mslong': 17809913453040846757}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-220998bd-94b4-4899-848b-14a09e1c3183', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:51.648 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-1b0fb202-a15f-4578-a079-413e846b84b2', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:51.659 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-5ce20252-7ee3-433a-afd4-97a83e0e1942', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:51.676 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:51.666562', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.666562', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10608329397487415044, 'uuid_mslong': 12641009152992233842}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-043efcd0-fec7-4d8c-b54c-f5c2f0bffb7e', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:51.765 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:51.751034', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.751034', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12665274423476574180, 'uuid_mslong': 14559706905692358710}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-2b1fdcaf-0532-49a7-89d0-ff8f0b532112', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:51.819 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-cceef24b-02aa-4ba9-8e77-159d86d7f996', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:51.831 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-9e20dffb-8801-43ec-8834-42aff3338aa0', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:51.860 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-01T01:57:51.847772', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.847772', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11156140797610135132, 'uuid_mslong': 437379043040970971}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-6e9b4c17-ad10-4551-8af6-958c601596a7', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:52.006 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0ce38428-da3d-46d3-af77-6621d179b916', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:52.041 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-4a3dbbdb-8ea8-4bb0-b802-e55592a31842', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:52.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:52.215 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-903126b7-3e5d-4c0d-a96c-0112009b6b6b', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:52.265 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-574ad662-aa66-4387-b6f3-ae5ecedfb9c0', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:52.415 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-65485163-b46b-4b7d-b4ea-ac3e45e48b1b', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:52.418 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-4053dbbf-3e1e-47e0-bc3a-9121239cf4a1', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:52.611 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b5885dd0-54b6-4de9-b985-e5729ac70731', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:52.643 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-c98a9e53-e0a4-47ea-a8a2-f875f5abdca0', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:52.796 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-be77af09-c6a1-4809-9837-02e4960fbc0f', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:52.806 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-8a8b9aea-1dd9-49ef-a3fa-731e869e0879', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:52.815 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-f824c559-dd9e-47f6-b43c-74bdb695c193', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:52.982 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-5978a90a-a2bb-4e72-b492-04055837bab6', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:52.985 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-cc861575-b1f8-4b01-8968-321ce8a165f7', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:53.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:53.148 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-8fd3efd9-9811-4d74-9c37-1d6a2f9d9316', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:53.164 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-d9273f8f-b94b-4a39-bdac-d445fbe24224', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:53.337 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-ea9f552e-98eb-4ef4-80bc-5642d6297bf3', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:53.344 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-e3763da9-84a2-42ee-81cf-a733fdda7b1f', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:53.529 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-00c0f81c-27b8-4bf7-b45d-8c60f682ff4c', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:53.532 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-61c5b9c6-20c7-4119-a5a5-c3f28a485fa3', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:53.535 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-fc63db98-24bf-4e4b-a4ef-4f72524f73db', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:53.689 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-3c1aef7d-30c2-406e-983f-8730edafccde', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:53.695 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-e4842649-b912-4285-b75d-22e7fdce5d46', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:53.865 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-257ec96c-0a05-47d5-8ed5-bceba5186da5', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:53.881 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-e087c260-1b7a-4195-aa63-9353258567c6', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:54.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:54.189 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:54.190 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:54.190 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:54.190 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:54.190 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:54.190 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:54.190 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:54.191 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.879 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-02593c6e-2575-40b8-a56f-63f0b1fa86fb', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:54.950 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-9a1cd17d-a4e0-4fed-a459-be82136524dd', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:54.954 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-4438a8df-7099-48db-8441-b15938d14eed', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:55.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:56.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:56.229 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-425b5a83-c87b-46df-8e2c-db3c3f271101', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:56.273 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-6c0bdafe-bb88-447a-b0d8-8cbade3c0907', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:57.472 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-f00a55d7-ce9e-43e2-9395-3dd8285232ff', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:57.533 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-3b996c4a-3334-4609-ac9b-97b4439f5e60', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:58.081 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:58.191 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:58.192 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:58.192 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:58.193 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:58.193 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:58.193 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:58.193 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:58.193 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:58.193 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.194 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:58.194 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.194 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.194 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:57:58.194 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.083 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:00.085 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:01.086 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:02.088 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:02.194 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:02.195 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:02.195 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:02.195 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:02.195 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:02.195 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:02.195 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:02.196 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:02.196 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.196 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:02.196 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.196 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.196 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:02.197 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.090 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:04.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:05.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:06.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:06.196 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:06.197 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:06.197 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:06.197 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:06.197 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:06.198 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:08.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:10.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:10.198 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:10.200 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:10.201 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.201 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:10.201 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.201 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.201 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:10.201 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:12.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:14.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:14.201 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:14.202 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:14.202 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:14.202 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:14.202 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:14.203 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:16.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:17.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:18.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:18.202 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:18.204 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.205 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:18.206 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.206 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.206 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:18.206 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:20.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:20.279 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'database_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:58:20.262077', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.262077', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11796017848418923458, 'uuid_mslong': 4531286052305127849}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3ee25ca8-2354-49a9-a3b3-dfbc332d73c2'}, 'oper': 'CREATE', 'request-id': 'req-89536832-5ade-4091-a3e7-dc5b51518fac', 'type': 'database_node', 'uuid': '3ee25ca8-2354-49a9-a3b3-dfbc332d73c2'} 09/01/2025 01:58:20.280 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.280 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.282 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { database_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4531286052305127849, "uuid_lslong": 11796017848418923458}, "enable": true, "created": "2025-09-01T01:58:20.262077", "last_modified": "2025-09-01T01:58:20.262077", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "3ee25ca8-2354-49a9-a3b3-dfbc332d73c2" } deleted = False >> 09/01/2025 01:58:20.282 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.291 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'database_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:58:20.280218', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.280218', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10071480984782399945, 'uuid_mslong': 378361382186796020}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '054035ac-25f8-43f4-8bc5-16983c0e69c9'}, 'oper': 'CREATE', 'request-id': 'req-1e468cdd-34f7-445d-b18b-bd3cca7149c2', 'type': 'database_node', 'uuid': '054035ac-25f8-43f4-8bc5-16983c0e69c9'} 09/01/2025 01:58:20.292 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.292 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.292 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { database_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 378361382186796020, "uuid_lslong": 10071480984782399945}, "enable": true, "created": "2025-09-01T01:58:20.280218", "last_modified": "2025-09-01T01:58:20.280218", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "054035ac-25f8-43f4-8bc5-16983c0e69c9" } deleted = False >> 09/01/2025 01:58:20.292 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.408 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:58:20.395054', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.395054', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12835641244198905252, 'uuid_mslong': 4848350493246179075}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4348cd16-0087-4703-b221-5bb498d425a4'}, 'oper': 'CREATE', 'request-id': 'req-67f78842-b75a-435a-8bb4-1eea81bd8c1f', 'type': 'database_node', 'uuid': '4348cd16-0087-4703-b221-5bb498d425a4'} 09/01/2025 01:58:20.408 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.409 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.409 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { database_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4848350493246179075, "uuid_lslong": 12835641244198905252}, "enable": true, "created": "2025-09-01T01:58:20.395054", "last_modified": "2025-09-01T01:58:20.395054", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "4348cd16-0087-4703-b221-5bb498d425a4" } deleted = False >> 09/01/2025 01:58:20.409 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:22.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:22.205 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:22.206 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:22.206 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:22.206 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:22.206 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:22.207 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:24.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:25.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:26.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:26.207 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:26.208 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:26.208 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.209 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.210 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:26.210 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:28.082 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:29.084 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.209 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:30.211 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:30.212 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:30.213 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:30.213 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:30.213 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:30.213 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:32.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:33.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.212 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:34.213 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:34.213 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:34.213 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:34.214 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:34.215 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:34.215 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:36.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:37.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:38.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:38.214 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:38.216 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:38.216 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:38.216 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:38.216 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.217 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.218 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:38.218 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:40.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:42.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:42.216 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:42.218 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:42.218 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:42.218 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:42.218 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:42.218 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:42.218 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:42.219 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:44.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:45.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:46.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:46.219 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:46.220 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:46.221 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:46.221 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:46.221 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:46.221 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:46.221 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:46.221 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:48.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:49.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:50.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:50.220 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:50.222 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:50.223 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:50.223 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:50.223 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:50.223 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:50.223 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:50.223 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:52.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.967 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:58:53.957876', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:53.957876', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13676038583231394089, 'uuid_mslong': 7839128237629393148}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6cca2d99-40a2-48fc-bdcb-0ca1f80b3129'}, 'oper': 'CREATE', 'request-id': 'req-f2c45455-00f1-4b72-b900-1a1e9bad38ff', 'type': 'analytics_node', 'uuid': '6cca2d99-40a2-48fc-bdcb-0ca1f80b3129'} 09/01/2025 01:58:53.968 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:53.968 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.968 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7839128237629393148, "uuid_lslong": 13676038583231394089}, "enable": true, "created": "2025-09-01T01:58:53.957876", "last_modified": "2025-09-01T01:58:53.957876", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6cca2d99-40a2-48fc-bdcb-0ca1f80b3129" } deleted = False >> 09/01/2025 01:58:53.969 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.987 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:58:53.973832', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:53.973832', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10557226880738800812, 'uuid_mslong': 4644487576518083042}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '407488d7-e213-41e2-9282-cde9d7a434ac'}, 'oper': 'CREATE', 'request-id': 'req-7052381b-b338-4995-aa45-2d649573240e', 'type': 'analytics_node', 'uuid': '407488d7-e213-41e2-9282-cde9d7a434ac'} 09/01/2025 01:58:53.988 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:53.988 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.988 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4644487576518083042, "uuid_lslong": 10557226880738800812}, "enable": true, "created": "2025-09-01T01:58:53.973832", "last_modified": "2025-09-01T01:58:53.973832", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "407488d7-e213-41e2-9282-cde9d7a434ac" } deleted = False >> 09/01/2025 01:58:53.989 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.082 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:58:54.061545', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:54.061545', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879227890580696674, 'uuid_mslong': 10756286823668662967}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '954601e0-7afc-42b7-96fa-c8203895ae62'}, 'oper': 'CREATE', 'request-id': 'req-c09260e8-71da-48cc-a61b-d3fea836d129', 'type': 'analytics_node', 'uuid': '954601e0-7afc-42b7-96fa-c8203895ae62'} 09/01/2025 01:58:54.082 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:54.083 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.086 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10756286823668662967, "uuid_lslong": 10879227890580696674}, "enable": true, "created": "2025-09-01T01:58:54.061545", "last_modified": "2025-09-01T01:58:54.061545", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "954601e0-7afc-42b7-96fa-c8203895ae62" } deleted = False >> 09/01/2025 01:58:54.086 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:54.223 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:54.224 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:54.225 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.225 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:54.225 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.225 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.225 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:54.225 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:56.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:58.082 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:58.225 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:58.226 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:58.226 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:58.226 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:58.226 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:58:58.227 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.085 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:00.086 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:01.088 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:02.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:02.226 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:02.228 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:02.229 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:02.229 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:02.229 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:02.229 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:02.229 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:02.229 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:04.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:05.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:06.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:06.228 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:06.229 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:06.229 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:06.230 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:06.231 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:06.231 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:08.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.721 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:09.711039', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:09.711039', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410244128994570297, 'uuid_mslong': 7346777207241002013}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '65f4fef1-f189-4c1d-8297-e72d7e6d3039'}, 'oper': 'CREATE', 'request-id': 'req-65902cdc-006d-4333-bc45-edc4870831f6', 'type': 'analytics_alarm_node', 'uuid': '65f4fef1-f189-4c1d-8297-e72d7e6d3039'} 09/01/2025 01:59:09.722 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:09.723 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.723 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_alarm_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7346777207241002013, "uuid_lslong": 9410244128994570297}, "enable": true, "created": "2025-09-01T01:59:09.711039", "last_modified": "2025-09-01T01:59:09.711039", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "65f4fef1-f189-4c1d-8297-e72d7e6d3039" } deleted = False >> 09/01/2025 01:59:09.723 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.103 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:10.086800', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:10.086800', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10503918820682058891, 'uuid_mslong': 906375348849557938}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0c941793-dd59-41b2-91c5-6a83e84b4c8b'}, 'oper': 'CREATE', 'request-id': 'req-5937b566-f775-43d5-a1c9-186dda6980e6', 'type': 'analytics_alarm_node', 'uuid': '0c941793-dd59-41b2-91c5-6a83e84b4c8b'} 09/01/2025 01:59:10.104 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:10.105 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.105 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_alarm_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 906375348849557938, "uuid_lslong": 10503918820682058891}, "enable": true, "created": "2025-09-01T01:59:10.086800", "last_modified": "2025-09-01T01:59:10.086800", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "0c941793-dd59-41b2-91c5-6a83e84b4c8b" } deleted = False >> 09/01/2025 01:59:10.105 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:10.194 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:10.172514', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:10.172514', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879587110886434360, 'uuid_mslong': 11828816951027650397}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a4286658-5f1f-4f5d-96fc-0ed5b74de638'}, 'oper': 'CREATE', 'request-id': 'req-5363ae59-4436-4596-ac85-f2751a51eb9c', 'type': 'analytics_alarm_node', 'uuid': 'a4286658-5f1f-4f5d-96fc-0ed5b74de638'} 09/01/2025 01:59:10.194 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:10.195 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.195 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_alarm_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11828816951027650397, "uuid_lslong": 10879587110886434360}, "enable": true, "created": "2025-09-01T01:59:10.172514", "last_modified": "2025-09-01T01:59:10.172514", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a4286658-5f1f-4f5d-96fc-0ed5b74de638" } deleted = False >> 09/01/2025 01:59:10.195 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.230 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:10.231 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.232 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:10.233 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.233 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.233 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:10.233 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:12.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:14.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:14.232 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:14.233 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:14.233 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:14.233 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:14.233 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:14.234 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:16.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:17.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:18.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:18.233 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:18.234 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.235 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.236 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:18.236 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:20.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.873 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:21.867078', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:21.867078', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13422430401004267031, 'uuid_mslong': 671617895336398686}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'}, 'oper': 'CREATE', 'request-id': 'req-0bcb5258-1518-4af3-bc32-66c2afda7f6a', 'type': 'analytics_snmp_node', 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'} 09/01/2025 01:59:21.874 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:21.874 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.874 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_snmp_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 671617895336398686, "uuid_lslong": 13422430401004267031}, "enable": true, "created": "2025-09-01T01:59:21.867078", "last_modified": "2025-09-01T01:59:21.867078", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "095210ea-dc52-4f5e-ba46-0d55aa436617" } deleted = False >> 09/01/2025 01:59:21.874 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:22.234 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:22.227040', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.227040', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10043001507123339914, 'uuid_mslong': 13054484782216790713}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'}, 'oper': 'CREATE', 'request-id': 'req-cc52ae74-0d4d-419c-a865-1eba05a95d49', 'type': 'analytics_snmp_node', 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'} 09/01/2025 01:59:22.235 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:22.235 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.236 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_snmp_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13054484782216790713, "uuid_lslong": 10043001507123339914}, "enable": true, "created": "2025-09-01T01:59:22.227040", "last_modified": "2025-09-01T01:59:22.227040", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "b52ad8be-69c6-42b9-8b5f-e8a91ecca28a" } deleted = False >> 09/01/2025 01:59:22.236 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.236 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:22.237 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:22.237 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:22.237 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:22.237 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:22.237 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:22.237 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:22.238 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.307 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:22.300301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.300301', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10295412184530401449, 'uuid_mslong': 11790706739729089604}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'}, 'oper': 'CREATE', 'request-id': 'req-79a7348d-975b-4e66-ac0e-d0c0e9ecf4b1', 'type': 'analytics_snmp_node', 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'} 09/01/2025 01:59:22.307 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:22.308 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.308 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_snmp_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11790706739729089604, "uuid_lslong": 10295412184530401449}, "enable": true, "created": "2025-09-01T01:59:22.300301", "last_modified": "2025-09-01T01:59:22.300301", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a3a1014f-585b-4444-8ee0-a6d59b29b8a9" } deleted = False >> 09/01/2025 01:59:22.308 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:24.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:25.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:26.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:26.237 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:26.239 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.240 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:26.240 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.240 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.240 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:26.240 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:28.084 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:29.086 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:30.087 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:30.239 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:30.240 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:30.240 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:30.240 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:30.240 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:30.240 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:30.240 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:30.241 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:32.091 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:33.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:34.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:34.240 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:34.242 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:34.242 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:34.242 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:34.242 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:34.242 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:34.242 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:34.243 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:36.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:37.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:38.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:38.242 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:38.244 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:38.245 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.245 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:38.245 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.245 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.245 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:38.245 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:40.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:42.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:42.244 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:42.245 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:42.245 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:42.245 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.245 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:42.246 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:44.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:45.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:46.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:46.246 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:46.247 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:46.248 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.248 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:46.248 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.248 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.248 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:46.248 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:48.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:49.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:50.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:50.248 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:50.249 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:50.249 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:50.249 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.249 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:50.249 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:50.249 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:50.250 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:52.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:53.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:54.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:54.250 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:54.251 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.252 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.253 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:54.253 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:56.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:58.085 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:58.252 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:58.253 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:58.253 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.254 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.255 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 01:59:58.255 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.087 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:00.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:01.091 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:02.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:02.254 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:02.256 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:02.256 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:02.256 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.256 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:02.256 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:02.256 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:02.257 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:04.096 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:06.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:06.257 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:06.258 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:06.258 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:06.258 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.258 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:06.258 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:06.259 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:08.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:10.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:10.258 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.260 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:10.261 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.261 7f5eef38bb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.261 7f5eef38bb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:10.261 7f5eef38bb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:12.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:14.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:14.261 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:14.262 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:14.262 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:14.262 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.262 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:14.262 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:14.262 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:14.263 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:16.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:17.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:18.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:18.263 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:18.264 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:18.264 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:18.264 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.265 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:18.266 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:20.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:21.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:22.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:22.265 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:22.267 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:22.267 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:22.267 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.267 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:22.267 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:22.267 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:22.268 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:24.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:25.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:26.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:26.267 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:26.269 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:26.269 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:26.269 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.270 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:26.270 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:26.270 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:26.270 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:26.270 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.271 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:26.271 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.271 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.271 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:26.271 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:28.086 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:29.088 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:30.090 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:30.270 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:30.271 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:30.271 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:30.271 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.271 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:30.271 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:30.271 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:30.272 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:32.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.096 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:34.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:34.272 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.274 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:34.275 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.275 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.275 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:34.275 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:36.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:37.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:38.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:38.274 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:38.275 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:38.275 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.276 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.277 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:38.277 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:40.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:42.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:42.276 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:42.277 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:42.277 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:42.277 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.277 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:42.277 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:42.277 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:42.278 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:44.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:45.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:46.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:46.278 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:46.279 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:46.279 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:46.279 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:46.280 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:48.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:49.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:50.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:50.280 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:50.282 7f5ef4c1c040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:50.282 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:50.282 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.283 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:50.283 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:50.283 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:50.283 7f5ef4c1c040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:50.283 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.284 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:50.284 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.284 7f5ef4c1c040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.284 7f5ef4c1c040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:50.284 7f5ef4c1c040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:52.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:53.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:54.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:54.283 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:54.285 7f5eef329f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:54.285 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:54.285 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.285 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:54.285 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:54.285 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:54.286 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:56.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:58.087 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:58.285 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:58.287 7f5eef329f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:58.287 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:58.287 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.288 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:58.288 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.288 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:58.288 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:58.288 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.289 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:58.289 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.289 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.289 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:00:58.289 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.088 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:00.090 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:01.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:02.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:02.288 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.290 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:02.291 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.291 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.291 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:02.291 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:04.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:05.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:06.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:06.291 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:06.292 7f5eef329f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:06.292 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:06.292 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.292 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:06.293 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:08.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:09.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:10.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:10.293 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:10.294 7f5eef329f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:10.294 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:10.294 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:10.295 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.478 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-d6447694-7a63-4e16-8ab1-f2203b502079', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 02:01:10.602 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T02:01:10.595372', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.595372', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11952340681741785153, 'uuid_mslong': 6586627194885784908}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.192'}, 'oper': 'CREATE', 'request-id': 'req-681edb8a-0de7-491e-8459-bf06930acfe8', 'type': 'virtual_router', 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841'} 09/01/2025 02:01:10.602 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:01:10.603 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.603 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = an-jenkins-deploy-platform-ansible-os-3995-2 elements = { display_name : "an-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6586627194885784908, "uuid_lslong": 11952340681741785153}, "enable": true, "created": "2025-09-01T02:01:10.595372", "last_modified": "2025-09-01T02:01:10.595372", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.192" } deleted = False >> 09/01/2025 02:01:10.603 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.941 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:01:10.925113', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.925113', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10735908003063543833, 'uuid_mslong': 17648289949859202616}}, 'parent_type': 'virtual-router', 'parent_uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:f4:eb:52:34:6a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.944 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:01:10.944 7f5eef329400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.944 7f5eef329400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17648289949859202616, "uuid_lslong": 10735908003063543833}, "enable": true, "created": "2025-09-01T02:01:10.925113", "last_modified": "2025-09-01T02:01:10.925113", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "f4eb5234-6a74-4638-94fd-9b728fe03819"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:f4:eb:52:34:6a"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:01:10.944 7f5eef329400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.966 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.972 7f5eef329f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.972 7f5eef329f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17648289949859202616, "uuid_lslong": 10735908003063543833}, "enable": true, "created": "2025-09-01T02:01:10.925113", "last_modified": "2025-09-01T02:01:10.956445", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7"}]uuid : "f4eb5234-6a74-4638-94fd-9b728fe03819"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:f4:eb:52:34:6a"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:01:10.972 7f5eef329f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:11.482 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-8b202558-6c4f-4e38-a46f-f0bfe3ec33be', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 02:01:12.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:14.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:14.295 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:14.297 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:14.297 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:14.297 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.297 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:14.297 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.297 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:14.298 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:16.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:17.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:18.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:18.298 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:18.299 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.300 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:18.300 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.300 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.300 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:18.300 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:20.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:21.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:22.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:22.299 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:22.301 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:22.301 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:22.301 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.301 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:22.301 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.301 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:22.302 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:24.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:25.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:26.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:26.302 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:26.303 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:26.303 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:26.303 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:26.304 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:28.088 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:29.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:30.091 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:30.304 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:30.305 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:30.305 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:30.305 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.305 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:30.305 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.305 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:30.306 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:32.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:33.096 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:34.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:34.305 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:34.308 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:34.308 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:34.308 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:34.309 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:36.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:37.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:38.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:38.309 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:38.312 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:38.312 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:38.312 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.312 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:38.312 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.312 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:38.313 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:40.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:42.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:42.313 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:42.314 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.315 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:42.315 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.315 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.315 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:42.315 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:44.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:45.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:46.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:46.315 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:46.316 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:46.316 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:46.316 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.316 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:46.316 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:46.317 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:48.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:49.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:50.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:50.317 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:50.318 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.319 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.320 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:50.320 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:52.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:53.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:54.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:54.319 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:54.320 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:54.320 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:54.320 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.320 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:54.320 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.320 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:54.321 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:56.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:58.089 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:58.321 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:58.322 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:58.322 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:58.322 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.322 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:01:58.323 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.090 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:00.091 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:01.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.322 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:02.324 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:02.325 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.325 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:02.325 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.325 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.325 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:02.325 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.992 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T02:02:02.985199', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:02.985199', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10627036196304171688, 'uuid_mslong': 16146534956283939804}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.18'}, 'oper': 'CREATE', 'request-id': 'req-7f54cc47-40c2-4259-85af-4adeb8e3ee0f', 'type': 'virtual_router', 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8'} 09/01/2025 02:02:02.992 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:02:02.993 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.993 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = an-jenkins-deploy-platform-ansible-os-3995-1 elements = { display_name : "an-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16146534956283939804, "uuid_lslong": 10627036196304171688}, "enable": true, "created": "2025-09-01T02:02:02.985199", "last_modified": "2025-09-01T02:02:02.985199", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.18" } deleted = False >> 09/01/2025 02:02:02.993 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:03.332 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:03.316910', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:03.316910', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12779270811540185455, 'uuid_mslong': 492471125198063140}}, 'parent_type': 'virtual-router', 'parent_uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:06:d5:9b:e1:0f']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.332 7f5eeffb10e0 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:02:03.333 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.333 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 492471125198063140, "uuid_lslong": 12779270811540185455}, "enable": true, "created": "2025-09-01T02:02:03.316910", "last_modified": "2025-09-01T02:02:03.316910", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "06d59be1-0f91-4624-b159-17187a5ee16f"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:06:d5:9b:e1:0f"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:02:03.333 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.358 7f5eeffb10e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.361 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.361 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 492471125198063140, "uuid_lslong": 12779270811540185455}, "enable": true, "created": "2025-09-01T02:02:03.316910", "last_modified": "2025-09-01T02:02:03.346223", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7"}]uuid : "06d59be1-0f91-4624-b159-17187a5ee16f"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:06:d5:9b:e1:0f"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:02:03.361 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:04.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:05.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:06.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:06.324 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:06.326 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:06.326 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.327 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.328 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:06.328 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:08.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:10.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:10.327 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:10.328 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:10.328 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:10.328 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.329 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:10.330 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.615 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:10.618 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:02:11.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:12.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:12.619 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:02:12.619 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:02:12.619 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:02:12.619 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:02:12.620 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:02:12.620 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:02:12.620 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:02:12.620 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:02:13.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:14.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:14.329 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:14.330 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:14.330 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:14.330 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.330 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:14.330 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.330 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:14.331 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:16.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:17.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:18.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:18.330 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:18.331 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.332 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.333 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:18.333 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:20.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:21.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:22.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:22.333 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:22.334 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:22.334 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:22.334 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:22.335 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.848 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:22.852 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:02:23.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:24.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:25.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:26.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:26.335 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:26.337 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:26.337 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:26.337 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.337 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:26.337 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.337 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:26.338 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:28.090 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:29.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:30.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:30.338 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:30.339 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:30.339 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:30.339 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.340 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:30.341 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:31.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:32.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:33.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:34.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:34.340 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:34.341 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:34.342 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.343 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.343 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:34.343 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:35.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:35.107 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:35.110 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:02:36.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:37.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:38.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:38.342 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:38.344 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:38.345 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.345 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:38.345 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.345 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.345 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:38.345 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:39.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:40.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:41.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:42.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:42.345 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:42.346 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:42.346 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:42.346 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.346 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:42.346 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.346 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:42.347 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:43.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:44.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:45.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:46.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:46.347 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:46.348 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:46.348 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:46.348 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.348 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:46.348 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.348 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:46.349 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:47.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:47.178 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:47.181 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:02:48.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:49.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:50.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:50.349 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:50.350 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:50.350 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:50.350 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.350 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:50.351 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:51.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:52.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:53.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:54.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:54.351 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:54.352 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.353 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:54.353 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.353 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.353 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:54.353 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:55.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:56.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:58.091 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:58.353 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:58.354 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:58.354 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:58.354 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.354 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:02:58.355 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:59.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:59.181 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:02:59.184 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:00.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:01.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:02.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:02.355 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:02.357 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:02.358 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.358 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:02.358 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.359 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.359 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:02.359 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:03.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:04.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:06.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:06.357 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:06.359 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:06.359 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:06.359 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.359 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:06.359 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.359 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:06.360 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:06.360 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.360 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:06.360 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.360 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.361 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:06.361 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:07.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:08.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:10.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:10.360 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:10.361 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:10.361 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:10.361 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:10.362 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:11.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:11.465 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:11.468 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:12.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:14.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:14.362 7f5eef329720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:14.364 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.365 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:14.365 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.365 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.365 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:14.365 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:15.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:16.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:17.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:18.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:18.364 7f5eef329720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:18.365 7f5eef329720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:18.366 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.367 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:18.367 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.367 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.367 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:18.367 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:19.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:20.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:21.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:22.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:22.367 7f5eef329720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:22.368 7f5eef329720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:22.368 7f5eef329720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:22.368 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.368 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:22.368 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.368 7f5eef329720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:22.369 7f5eef329720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:23.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:23.520 7f5eef329720 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:23.523 7f5eef329720 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:24.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:25.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:26.155 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:26.369 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:26.370 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:26.370 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:26.370 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:26.371 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.092 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:29.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:30.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:30.371 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:30.372 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:30.372 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:30.372 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.372 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:30.373 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:31.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:32.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:33.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:34.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:34.372 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:34.373 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:34.374 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.375 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.375 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:34.375 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:35.739 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:35.743 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:36.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:37.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:38.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:38.374 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:38.375 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:38.375 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:38.375 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.375 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:38.376 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:39.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:40.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:42.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:42.376 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:42.377 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:42.377 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:42.377 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.377 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:42.377 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.377 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:42.378 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:43.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:44.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:45.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:46.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:46.378 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:46.379 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:46.379 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:46.379 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.379 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:46.379 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.379 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:46.380 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:47.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:47.908 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:47.911 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:03:48.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:49.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:50.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:50.380 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:50.382 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:50.382 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:50.382 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.382 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:50.382 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.382 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:50.383 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:51.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:52.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:53.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:54.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:54.383 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:54.384 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:54.385 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.385 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:54.385 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.385 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.385 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:54.385 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:55.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:56.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:58.093 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:58.385 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:58.387 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:58.387 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:58.387 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:03:58.388 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:59.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:59.954 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:03:59.957 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:00.096 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:01.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:02.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:02.388 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:02.390 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:02.390 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:02.390 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.390 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:02.390 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.390 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:02.391 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:03.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:04.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:06.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:06.391 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:06.392 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:06.392 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:06.392 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.392 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:06.392 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:06.393 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:07.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:08.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:10.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:10.393 7f5eef329860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:10.394 7f5eef329860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:10.394 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:10.394 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.394 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:10.394 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.394 7f5eef329860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:10.395 7f5eef329860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:11.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:12.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:12.241 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:12.244 7f5eef329860 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:13.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:14.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:04:14.195 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:04:14.395 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:14.396 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:14.397 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.398 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.398 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:14.398 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:15.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:16.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:17.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:18.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:18.397 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:18.398 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:18.398 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:18.398 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.398 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:18.399 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:19.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:20.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:21.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:22.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:22.399 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:22.401 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:22.402 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.402 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:22.402 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.402 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.402 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:22.402 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:24.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:24.280 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:24.283 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:25.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:26.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:26.401 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:26.403 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:26.403 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:26.403 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.403 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:26.403 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.403 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:26.404 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.094 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:29.096 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:30.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:30.403 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:30.404 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.405 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:30.406 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.406 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.406 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:30.406 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:32.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:34.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:34.405 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:34.407 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.408 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:34.408 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.408 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.408 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:34.408 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:36.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:36.608 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:36.611 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:37.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:38.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:38.408 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.410 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:38.411 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.411 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.411 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:38.411 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:40.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:42.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:42.411 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:42.412 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.413 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:42.413 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.413 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.413 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:42.413 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:44.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:45.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:46.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:46.413 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:46.415 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:46.415 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:46.415 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.415 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:46.415 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.415 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:46.416 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:48.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:49.017 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:04:49.021 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:04:49.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:50.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:50.416 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:50.417 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:50.418 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.419 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:50.419 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.419 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.419 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:50.419 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:52.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:53.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:54.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:54.418 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:54.420 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:54.420 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:54.420 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:54.421 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.422 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.422 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:54.422 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:56.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:58.095 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:58.421 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:58.422 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:58.423 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:58.423 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.423 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:58.423 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.423 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:04:58.424 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:00.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:01.082 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:01.085 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:01.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:02.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:02.424 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:02.425 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:02.425 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:02.425 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.425 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:02.425 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:02.426 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:04.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:05.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:06.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:06.426 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:06.428 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:06.428 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:06.428 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.428 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:06.428 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.428 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:06.429 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:06.429 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.429 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:06.429 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.429 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.430 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:06.430 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:08.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:10.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:10.429 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:10.430 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:10.431 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.432 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.432 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:10.432 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:12.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.737 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:13.741 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:14.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:14.431 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:14.432 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:14.433 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.434 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:14.434 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.434 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.434 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:14.434 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.111 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:05:15.111 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:05:15.111 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:05:15.111 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:05:15.111 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:05:15.112 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:05:15.112 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:05:15.112 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:05:15.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:16.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:17.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:18.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:18.433 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:18.434 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.435 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.436 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:18.436 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:20.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:21.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:22.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:22.436 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:22.437 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:22.437 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:22.437 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.438 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:22.439 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:24.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.767 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:25.770 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:26.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:26.438 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:26.440 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:26.440 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:26.440 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:26.441 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.442 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.442 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:26.442 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.096 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:29.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:30.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:30.442 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:30.443 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:30.444 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.444 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:30.444 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.444 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.444 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:30.444 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:32.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:34.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:34.443 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:34.445 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:34.446 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.447 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:34.447 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.447 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.447 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:34.447 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:36.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:37.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:38.036 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:38.040 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:38.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:38.446 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:38.447 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.448 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:38.449 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.449 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.449 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:38.449 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:40.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:42.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:42.448 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:42.449 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:42.449 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:42.450 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.450 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:42.450 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.450 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:42.450 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:42.451 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.451 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:42.451 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.451 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.451 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:42.451 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:44.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:45.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:46.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:46.450 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:46.451 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:46.452 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.453 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:46.453 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.453 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.453 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:46.453 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:48.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:49.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:50.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:50.272 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:05:50.276 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:05:50.452 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:50.453 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:50.453 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:50.453 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.453 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:50.453 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:50.454 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:52.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:53.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:54.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:54.454 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:54.455 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:54.456 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.457 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.457 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:54.457 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:56.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:58.097 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:58.456 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:58.457 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:58.457 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.458 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:05:58.459 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:00.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:01.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:02.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:02.458 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:02.459 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:02.459 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.460 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:02.461 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.606 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:02.609 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:03.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:04.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:06.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:06.461 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:06.462 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.463 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.463 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:06.463 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:08.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:09.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:10.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:10.462 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:10.463 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:10.464 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.465 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.465 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:10.465 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:12.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:14.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:14.465 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:14.466 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:14.466 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:14.466 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:14.467 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.031 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:15.034 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:15.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:16.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:18.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:18.467 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:18.468 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:18.468 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:18.468 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.468 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:18.468 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.468 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:18.469 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:20.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:22.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:22.469 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:22.470 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:22.471 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.472 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.472 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:22.472 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:24.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:26.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:26.471 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:26.473 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:26.473 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:26.473 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.473 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:26.473 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.473 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:26.474 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.289 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:27.292 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:28.098 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:29.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:30.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:30.474 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:30.475 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:30.475 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:30.475 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.475 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:30.475 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.475 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:30.476 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:32.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:34.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:34.475 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:34.476 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:34.477 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.478 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.478 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:34.478 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:36.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:37.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:38.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:38.477 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:38.479 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.480 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.480 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:38.480 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:39.451 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:39.453 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:40.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:42.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:42.480 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:42.481 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:42.481 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:42.481 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.481 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:42.481 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.481 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:42.482 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:44.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:45.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:46.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:46.482 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:46.483 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:46.483 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:46.483 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:46.484 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:48.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:49.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:50.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:50.484 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:50.485 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:50.485 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:50.485 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.485 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:50.485 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.485 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:50.486 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:51.453 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:06:51.456 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:06:52.153 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:53.155 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:54.157 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:54.485 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:54.487 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.488 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:54.488 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.488 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.488 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:54.488 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.158 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:56.159 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:58.099 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:58.488 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:58.489 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:58.490 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:58.490 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.490 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:58.490 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.490 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:58.490 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:58.491 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.491 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:58.491 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.491 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.491 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:06:58.491 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.101 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:00.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:01.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:02.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:02.491 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:02.492 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:02.492 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:02.492 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:02.493 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:03.520 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:03.524 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:04.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:05.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:06.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:06.493 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:06.495 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:06.495 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:06.495 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.495 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:06.496 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.496 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:06.496 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:06.496 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.497 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:06.497 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.497 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.497 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:06.497 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:08.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:09.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:10.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:10.495 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:10.497 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:10.498 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.498 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:10.498 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.498 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.498 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:10.498 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:12.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:14.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:14.498 7f5eef329900 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:14.499 7f5eef329900 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:14.499 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:14.499 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.499 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:14.499 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.499 7f5eef329900 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:14.500 7f5eef329900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.112 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:16.115 7f5eef329900 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:16.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.717 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:07:16.718 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:07:17.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:18.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:18.500 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:18.501 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:18.501 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:18.501 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:18.502 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.503 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.503 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:18.503 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:20.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:21.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:22.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:22.502 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:22.503 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:22.503 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:22.503 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.504 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:22.505 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:24.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:25.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:26.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:26.504 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:26.506 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:26.507 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.507 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:26.507 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.507 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.507 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:26.507 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.100 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:28.244 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:28.247 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:29.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:30.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:30.507 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:30.508 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:30.508 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:30.508 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.508 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:30.508 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:30.509 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:32.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:33.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:34.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:34.509 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:34.511 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:34.511 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:34.511 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.511 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:34.511 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.511 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:34.512 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:36.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:37.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:38.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:38.511 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:38.513 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:38.513 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:38.513 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.513 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:38.513 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.513 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:38.514 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:40.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:40.262 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:40.265 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:41.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:42.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:42.514 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:42.515 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:42.515 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:42.515 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:42.516 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:44.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:45.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:46.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:46.516 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:46.517 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:46.518 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.519 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.519 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:46.519 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:48.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:49.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:50.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:50.518 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:50.520 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:50.520 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:50.520 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.521 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.522 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:50.522 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:52.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:52.603 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:07:52.606 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:07:53.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:54.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:54.521 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:54.522 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:54.523 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:54.523 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.523 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:54.523 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.523 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:54.524 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:56.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:58.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:58.524 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:58.525 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:58.525 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:58.525 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.525 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:07:58.526 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:00.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:01.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:02.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:02.526 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:02.528 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:02.528 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:02.528 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.529 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:02.529 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.529 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:02.529 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:02.529 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.529 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:02.530 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.530 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.530 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:02.530 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.730 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:04.734 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:05.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:06.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:06.529 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:06.530 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:06.530 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:06.530 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.530 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:06.531 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:08.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:09.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:10.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:10.531 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:10.532 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:10.532 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:10.532 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.532 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:10.532 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:10.533 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:12.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:13.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:14.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:14.533 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:14.534 7f5eef329d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:14.534 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:14.534 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.534 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:14.534 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:14.535 7f5eef329d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:16.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.182 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:17.185 7f5eef329d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:08:17.574 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:08:18.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:18.535 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:18.536 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:18.537 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.538 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:18.538 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.538 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.538 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:18.538 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:20.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:21.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:22.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:22.538 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:22.540 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:22.540 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:22.540 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.541 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.542 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:22.542 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:24.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:25.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:26.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:26.542 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:26.543 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:26.543 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:26.543 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.543 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:26.543 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.543 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:26.544 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.102 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.174 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:29.177 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:30.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:30.544 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:30.545 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.546 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:30.546 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.546 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.546 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:30.546 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:32.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:33.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:34.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:34.545 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:34.546 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.547 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.548 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:34.548 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:36.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:37.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:38.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:38.547 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:38.549 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:38.549 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:38.549 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.549 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:38.549 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.549 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:38.550 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:40.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.352 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:41.355 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:42.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:42.550 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:42.552 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.553 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:42.553 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.553 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.553 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:42.553 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:44.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:45.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:46.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:46.553 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:46.554 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:46.554 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:46.554 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.554 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:46.554 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.554 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:46.555 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:48.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:49.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:50.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:50.555 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:50.557 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:50.557 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:50.557 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.557 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:50.557 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.557 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:50.558 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:52.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.613 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:08:53.616 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:08:54.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:54.557 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:54.558 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:54.558 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.559 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:54.560 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:56.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:58.103 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:58.560 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:58.561 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:58.561 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:58.561 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.562 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.563 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:08:58.563 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:00.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:01.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:02.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:02.562 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:02.564 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:02.565 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.565 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:02.565 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.565 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.565 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:02.565 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:04.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.636 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:05.639 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:06.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:06.565 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:06.566 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:06.566 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:06.566 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.566 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:06.567 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:08.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:10.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:10.568 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:10.569 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:10.569 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:10.569 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.569 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:10.569 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.569 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:10.570 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:12.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:13.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:14.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:14.570 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:14.571 7f5eef3295e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:14.571 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:14.571 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.572 7f5eef3295e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:14.573 7f5eef3295e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:16.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.964 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:17.967 7f5eef3295e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:18.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:18.572 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:18.574 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:18.574 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:18.574 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.574 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:18.574 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.574 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:18.575 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:20.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:21.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:22.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:22.574 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:22.576 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:22.577 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.577 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:22.577 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.577 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.577 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:22.577 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:24.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:25.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:26.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:26.577 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:26.578 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:26.578 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:26.578 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.578 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:26.578 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.578 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:26.579 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.104 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.106 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.997 7f5eef329ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:30.000 7f5eef329ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:30.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:30.579 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:30.580 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:30.580 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:30.580 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:30.581 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:32.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:33.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:34.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:34.581 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:34.582 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:34.582 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:34.582 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.582 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:34.582 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.582 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:34.583 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:36.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:37.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:38.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:38.582 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:38.584 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:38.585 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.585 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:38.585 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.585 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.585 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:38.585 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:40.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:41.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:42.106 7f5eef329ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:42.109 7f5eef329ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:42.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:42.585 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:42.586 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:42.586 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:42.586 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:42.587 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:44.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:45.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:46.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:46.587 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:46.589 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:46.589 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:46.589 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.589 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:46.589 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.589 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:46.590 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:48.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:49.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:50.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:50.590 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:50.592 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:50.592 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:50.592 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.592 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:50.592 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.592 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:50.593 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:52.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:53.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:54.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:54.344 7f5eef329ea0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:09:54.347 7f5eef329ea0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:09:54.593 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:54.594 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:54.594 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:54.594 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.595 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:54.595 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.595 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:54.595 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:54.595 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.596 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:54.596 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.596 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.596 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:54.596 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:56.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:58.105 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:58.595 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:58.597 7f5eef329ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:58.597 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:58.597 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.597 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:58.597 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.597 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:09:58.598 7f5eef329ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.987 7f5eef3b67c0 [contrail-api] [ERROR]: Failed to resync: Unable to establish connection to http://10.0.0.249:5000/v3/domains? 09/01/2025 02:10:00.988 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.988 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:00.988 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:01.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:02.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:02.598 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:02.600 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:02.600 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:02.600 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.601 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:02.602 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:04.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:05.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:06.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:06.600 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:06.602 7f5eef3b6360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.603 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:06.604 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.604 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.604 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:06.604 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.605 7f5eef3b6360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:07.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:08.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:09.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.604 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:10.605 7f5eef3b6360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:10.605 7f5eef3b6360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.606 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:10.607 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:12.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:13.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:14.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:14.607 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:14.608 7f5eef3b6360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:14.608 7f5eef3b6360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:14.608 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.608 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:14.608 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.608 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:14.609 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:16.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:17.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:18.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:18.609 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:18.610 7f5eef3b6360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:18.610 7f5eef3b6360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:18.610 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.610 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:18.610 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.610 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:18.611 7f5eef3b6360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.120 7f5eef3b6360 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:19.123 7f5eef3b6360 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:19.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:19.173 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:10:19.174 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:10:20.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:21.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:22.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:22.611 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:22.612 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:22.613 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.614 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:22.614 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.614 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.614 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:22.614 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:24.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:25.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:26.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:26.613 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.615 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:26.616 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.616 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.616 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:26.616 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:29.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.616 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:30.617 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.618 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:30.619 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.619 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.619 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:30.619 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:31.236 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:31.239 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:32.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:33.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.619 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:34.620 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:34.620 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:34.620 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:34.621 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.622 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.622 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:34.622 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:36.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:37.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.622 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:38.623 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:38.623 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:38.623 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.623 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:38.623 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.623 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:38.624 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:40.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:41.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.625 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:42.626 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:42.626 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:42.626 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.626 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:42.626 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.626 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:42.627 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:43.674 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:43.677 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:44.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:45.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:46.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:46.627 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:46.629 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:46.629 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:46.629 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.629 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:46.630 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:48.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:49.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:50.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:50.630 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:50.631 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.632 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:50.633 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.633 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.633 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:50.633 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:52.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:53.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:54.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:54.633 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:54.634 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.635 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:54.635 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.635 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.635 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:54.635 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.153 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:55.758 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:10:55.761 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:10:56.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.107 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.634 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:58.635 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:10:00.987869] connection down >>, ] >> 09/01/2025 02:10:58.636 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.637 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.637 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:10:58.637 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:00.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:01.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:01.308 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:01.309 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:01.309 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:02.636 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:02.637 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:02.638 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.639 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:02.639 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.639 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.639 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:02.639 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:04.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:05.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.638 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:06.686 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:06.686 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:06.686 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:06.687 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:08.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:08.133 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:08.137 7f5ef4bd4540 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:09.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.686 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.688 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:10.689 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.689 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.689 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:10.689 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:12.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:13.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.689 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:14.690 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:14.690 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:14.690 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:14.691 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:16.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:17.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:18.140 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:18.690 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:18.691 7f5ef4bd4540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:18.691 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.692 7f5ef4bd4540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.693 7f5ef4bd4540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:18.693 7f5ef4bd4540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:11:20.089 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:11:20.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:20.549 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:20.552 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:21.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:22.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:22.692 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:22.694 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.695 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:22.695 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.695 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.695 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:22.695 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:24.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:25.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.153 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.694 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:26.696 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.697 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:26.697 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.697 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.697 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:26.697 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.108 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:29.110 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.697 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:30.698 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:30.698 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:30.698 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.698 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:30.698 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.698 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:30.699 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.807 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:32.810 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:33.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.699 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:34.701 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:34.701 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:34.701 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.701 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:34.701 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.701 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:34.702 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:36.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:37.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.702 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:38.703 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:38.703 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:38.703 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.703 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:38.703 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.703 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:38.704 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:40.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:41.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.703 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:42.704 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.705 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:42.706 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.706 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.706 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:42.706 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:44.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:45.012 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:45.015 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:45.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:46.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:46.705 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:46.707 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:46.707 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.708 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:46.709 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.709 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.709 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:46.709 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:48.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:49.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:50.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:50.708 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:50.710 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.711 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:50.711 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.711 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.711 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:50.711 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:52.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:53.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:54.155 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:54.710 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:54.711 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:54.711 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:54.711 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.711 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:54.712 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.157 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:56.158 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:57.024 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:11:57.028 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:11:58.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:58.712 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:58.714 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:58.714 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:58.714 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.714 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:58.714 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.714 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:11:58.715 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:00.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:01.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.716 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:02.717 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.718 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:02.719 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.719 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.719 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:02.719 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:04.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:05.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.718 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:06.720 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:06.720 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:06.720 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.720 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:06.720 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.720 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:06.721 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:08.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.340 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:09.343 7f5ef4bd45e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:10.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:10.721 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:10.722 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:10.722 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:10.722 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:10.723 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:12.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:13.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.723 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:14.724 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:14.724 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:14.724 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.724 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:14.724 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.724 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:14.725 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:16.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:17.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:18.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:18.725 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:18.727 7f5ef4bd45e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:18.727 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:18.727 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.727 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:18.727 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.727 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:18.728 7f5ef4bd45e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:21.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:21.897 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:21.900 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:22.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:22.727 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:22.728 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.729 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.730 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:22.730 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:24.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:25.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:26.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:26.730 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:26.731 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:26.731 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:26.731 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.731 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:26.731 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.731 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:26.732 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.109 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:29.111 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.732 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:30.733 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:30.733 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:30.733 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.733 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:30.733 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.733 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:30.734 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:32.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.953 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:33.957 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:34.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:34.734 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:34.735 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:34.735 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:34.735 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.735 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:34.735 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.735 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:34.736 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:36.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:37.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.736 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:38.778 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:38.779 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.780 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.780 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:38.780 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:40.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:41.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.779 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:42.780 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:42.780 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:42.780 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:42.781 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:44.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:45.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:46.069 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:46.072 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:46.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:46.781 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:46.783 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.784 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:46.784 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.784 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.784 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:46.784 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:48.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:49.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:50.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:50.784 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:50.786 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:50.786 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:50.786 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.786 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:50.786 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.786 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:50.787 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:52.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:53.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:54.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:54.787 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:54.788 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:54.788 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:54.788 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.788 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:54.788 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.788 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:54.789 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.155 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:56.156 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.346 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:12:58.350 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:12:58.789 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:58.791 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:58.792 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:58.792 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.792 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:58.792 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.792 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:58.792 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:58.793 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.793 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:58.793 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.793 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.793 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:12:58.793 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:00.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:01.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:02.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:02.792 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:02.794 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:02.794 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:02.794 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.794 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:02.794 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:02.795 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:04.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:05.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.795 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:06.796 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:06.796 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.797 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:06.798 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.798 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.798 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:06.798 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:08.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:09.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.573 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:10.576 7f5ef4bc0040 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:10.797 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:10.799 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.800 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:10.800 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.800 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.800 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:10.800 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:12.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:13.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.800 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:14.801 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:14.801 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:14.801 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.801 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:14.802 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:16.144 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:17.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:18.148 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:18.801 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:18.803 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.804 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:18.804 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.804 7f5ef4bc0040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.804 7f5ef4bc0040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:18.804 7f5ef4bc0040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:20.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.153 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:13:21.699 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:13:21.700 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:13:22.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:22.804 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:22.805 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:22.805 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:22.805 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:22.806 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.134 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:23.137 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:23.155 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:24.156 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:25.158 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.160 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.806 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:26.808 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:26.808 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:26.808 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.809 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:26.809 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.809 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:26.809 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:26.809 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.809 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:26.810 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.810 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.810 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:26.810 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.112 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:29.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.115 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.809 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:30.811 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:30.811 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:30.811 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.811 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:30.811 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.811 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:30.812 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:32.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:33.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.812 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:34.813 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:34.814 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.815 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:34.815 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.815 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.815 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:34.815 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:35.320 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:35.324 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:36.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:37.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.814 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:38.816 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:38.816 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:38.816 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.816 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:38.816 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.817 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:38.818 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:40.172 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:41.173 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.174 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.816 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:42.818 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:42.819 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.820 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.820 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:42.820 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.175 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:44.176 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:45.177 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:46.179 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:46.820 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:46.822 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:46.822 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:46.822 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.822 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:46.822 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.822 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:46.823 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.180 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:47.598 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:47.602 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:13:48.182 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:49.184 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:50.185 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:50.824 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:50.825 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:50.825 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:50.825 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:50.826 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.827 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.827 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:50.827 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.187 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:52.189 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:53.190 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:54.192 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:54.826 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:54.827 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:54.827 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:54.827 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:54.828 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.193 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:56.194 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.113 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.828 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:58.830 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.831 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:58.831 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.831 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.831 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:13:58.831 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:59.648 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:13:59.651 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:00.116 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:01.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.119 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.831 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:02.832 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:02.832 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:02.832 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.832 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:02.832 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:02.833 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.121 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:04.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:05.124 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.833 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:06.834 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:06.834 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:06.834 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.834 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:06.834 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.834 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:06.835 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:08.128 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:09.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.132 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.835 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:10.836 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:10.836 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:10.836 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:10.837 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:11.617 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:11.620 7f5ef4bd4220 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:12.134 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:13.136 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.138 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.837 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:14.839 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:14.839 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:14.839 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.839 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:14.839 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.839 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:14.840 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:16.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:17.142 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:18.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:18.840 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:18.842 7f5ef4bd4220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.843 7f5ef4bd4220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.844 7f5ef4bd4220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:18.844 7f5ef4bd4220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:20.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:21.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:22.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:22.570 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:14:22.570 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:14:22.570 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:14:22.570 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:14:22.571 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:14:22.571 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:14:22.571 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:14:22.571 7f5eef38be00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:14:22.844 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:22.845 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:22.845 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:22.846 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.846 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:22.846 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.846 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:22.846 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:22.846 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.847 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:22.847 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.847 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.847 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:22.847 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.150 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:24.151 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:24.186 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:24.188 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:25.152 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.154 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.198 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:26.201 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:26.846 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:26.848 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.849 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:26.849 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.849 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.849 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:26.849 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.114 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:28.194 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:28.196 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:29.117 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.118 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.849 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:30.850 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:30.850 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:30.850 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:30.851 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.120 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:32.122 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:33.123 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.125 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.852 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:34.853 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:34.853 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:34.853 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.853 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:34.853 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.853 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:34.854 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.126 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:36.127 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:37.129 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.130 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.854 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:38.856 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:38.856 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:38.856 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.856 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:38.856 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.856 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:38.857 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.131 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:40.133 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:41.135 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.137 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.857 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:42.858 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:42.859 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:42.859 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.859 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:42.859 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.859 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:42.859 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:42.860 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.860 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:42.860 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.860 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.860 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:42.860 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.139 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:44.141 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:45.143 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:45.799 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:14:45.802 7f5ef4bd42c0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:14:46.145 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:46.860 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:46.861 7f5ef4bd42c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.862 7f5ef4bd42c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.863 7f5ef4bd42c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794152634 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.131:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 09/01/2025 02:14:46.863 7f5ef4bd42c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.146 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:48.147 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:49.149 7f5eef38b5e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.