09/01/2025 01:56:34.145 7f0f41b52a40 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:56:34.145 7f0f41b52a40 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:56:34.145 7f0f41b52a40 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:56:34.146 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.249:2181,10.0.0.38:2181,10.0.0.131:2181 09/01/2025 01:56:34.146 7f0f41b52a40 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:56:34.147 7f0f3d06a9a0 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:56:34.147 7f0f3d06a9a0 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:56:34.147 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:56:34.147 7f0f3d06ad60 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:56:34.147 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:56:34.147 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794147667 collector_name = collector_ip = collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.38:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:56:34.148 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.165 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.165 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:34.165 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.152 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:38.152 7f0f3d06ad60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:38.153 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:38.154 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:38.154 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:38.154 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:38.154 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:38.154 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:38.155 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.155 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:38.155 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.155 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.155 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794147667 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:38.155 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.155 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:42.156 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:42.157 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.157 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:42.157 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.157 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.157 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794147667 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:42.157 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.160 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:46.161 7f0f3d06ad60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:46.161 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:46.161 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:46.161 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:46.161 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794147667 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:46.162 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.163 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:50.164 7f0f3d06ad60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:50.164 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:50.164 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:50.164 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:50.164 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:50.164 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794147667 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:50.165 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.167 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:54.168 7f0f3d06ad60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:54.168 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:54.168 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:54.168 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:54.168 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:54.168 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794147667 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.249:8086, 10.0.0.131:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:54.169 7f0f3d06ad60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.537 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.249:9041', '10.0.0.38:9041', '10.0.0.131:9041'] 09/01/2025 01:56:54.537 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:56:54.537 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:56:54.537 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:56:54.538 7f0f3d06ae00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.538 7f0f3d06ae00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.538 7f0f3d06ae00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.540 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:56:54.580 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:56:54.583 7f0f3d2bf680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.583 7f0f3d2bf680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.583 7f0f3d2bf680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.603 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:56:54.604 7f0f3d2bf680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.604 7f0f3d2bf680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.604 7f0f3d2bf680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.610 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:56:54.684 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:56:54.685 7f0f3dd5e540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.685 7f0f3dd5e540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.685 7f0f3dd5e540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.823 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:56:54.833 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:56:54.834 7f0f3d2bfd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.834 7f0f3d2bfd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.834 7f0f3d2bfd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.834 7f0f3d2bfd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.834 7f0f3d2bfd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.834 7f0f3d2bfd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.842 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:56:54.842 7f0f3d2bfd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.842 7f0f3d2bfd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.842 7f0f3d2bfd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.870 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:54.870 7f0f3dcd80e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.871 7f0f3dcd80e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.871 7f0f3dcd80e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.873 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:54.873 7f0f3dcd80e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.873 7f0f3dcd80e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.873 7f0f3dcd80e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.874 7f0f3dcd8400 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:54.874 7f0f3dcd8680 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.874 7f0f3dcd8680 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.874 7f0f3dcd8680 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.877 7f0f3dcd8400 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:54.877 7f0f3dcd8900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.877 7f0f3dcd8900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.877 7f0f3dcd8900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.878 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config']: Unknown id: global_system_config default-global-system-config 09/01/2025 01:56:54.920 7f0f41b52a40 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-vrouter-config']: Unknown id: global_vrouter_config default-global-system-config:default-global-vrouter-config 09/01/2025 01:57:06.929 7f2a3d611fc0 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:57:06.929 7f2a3d611fc0 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:57:06.929 7f2a3d611fc0 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:57:06.930 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.249:2181,10.0.0.38:2181,10.0.0.131:2181 09/01/2025 01:57:06.930 7f2a3d611fc0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:57:06.931 7f2a3985a860 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:57:06.931 7f2a3985a860 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:57:06.931 7f2a3985ac20 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:57:06.931 7f2a3985ac20 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:57:06.932 7f2a3985ac20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.942 7f2a3985ac20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.942 7f2a3985ac20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:06.942 7f2a3985ac20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.973 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.249:9041', '10.0.0.38:9041', '10.0.0.131:9041'] 09/01/2025 01:57:06.973 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:57:06.973 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:57:06.973 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:57:06.974 7f2a3985acc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.974 7f2a3985acc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:06.974 7f2a3985acc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.978 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:57:07.009 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:57:07.011 7f2a38d07540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.011 7f2a38d07540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:07.011 7f2a38d07540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.021 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:57:07.021 7f2a39836f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.021 7f2a39836f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:07.022 7f2a39836f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.028 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:57:07.064 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:57:07.064 7f2a39807400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.064 7f2a39807400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:07.064 7f2a39807400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.171 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:57:07.181 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:57:07.182 7f2a38d07c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.183 7f2a38d07c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:07.183 7f2a38d07c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.183 7f2a38d07c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.183 7f2a38d07c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:07.183 7f2a38d07c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.186 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:57:07.187 7f2a38d07c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.187 7f2a38d07c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:07.187 7f2a38d07c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.243 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:57:07.243 7f2a38cf0720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.244 7f2a38cf0720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:07.244 7f2a38cf0720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.247 7f2a3d611fc0 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:57:07.247 7f2a38cf0720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.248 7f2a38cf0720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:07.248 7f2a38cf0720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.249 7f2a38cf0a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:57:07.250 7f2a38cf0cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.250 7f2a38cf0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:07.250 7f2a38cf0cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.254 7f2a38cf0a40 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:57:07.255 7f2a38cf0f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.255 7f2a38cf0f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:07.255 7f2a38cf0f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.508 7f2a38cf0860 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.249_0 09/01/2025 01:57:07.509 7f2a38cf0860 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly) 09/01/2025 01:57:07.522 7f2a38cf0860 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (nightly) 09/01/2025 01:57:07.528 7f2a38cf0860 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running walk 09/01/2025 01:57:07.574 7f2a38cf0860 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 09/01/2025 01:57:07.720 7f2a38155540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.720 7f2a38155540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:07.721 7f2a38155540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.726 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting consumer greenlet 09/01/2025 01:57:07.726 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:57:07.726 7f2a380eb900 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting publisher greenlet 09/01/2025 01:57:07.726 7f2a380eb900 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Publisher connection 09/01/2025 01:57:07.728 7f2a3d611fc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-1..dummy 09/01/2025 01:57:07.728 7f2a3d611fc0 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 01:57:07.729 7f2a38155540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.729 7f2a38155540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ConfigApiWorkerTrace: data = << name = worker-0 worker_id = 0 introspect_port = 8084 admin_port = 8095 introspect_url = http://10.0.0.249:8084 >> 09/01/2025 01:57:07.729 7f2a38155540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.736 7f2a380eb900 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 09/01/2025 01:57:07.737 7f2a380eb7c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:57:07.742 7f2a38cf0860 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Time elapsed in resyncing db: 0:00:00.233270 09/01/2025 01:57:07.743 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-627a4ee1-7b6b-4f96-97ad-6e4314a60029', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:57:07.759 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:57:07.760 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-d73327dd-fcdc-48c2-9ec5-f2dafab03c46', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:57:07.761 7f2a38cf0860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.761 7f2a38cf0860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:57:07.333259", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:57:07.761 7f2a38cf0860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.765 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-34de5f60-0dc2-44ba-9d39-a1c3bc2a7184', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:57:07.769 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/01/2025 01:57:07.769 7f2a33db02c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.770 7f2a33db02c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:57:07.735619", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:57:07.770 7f2a33db02c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.777 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-1..dummy'] 09/01/2025 01:57:07.797 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:57:07.800 7f2a380eb7c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:57:07.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-1..dummy'] 09/01/2025 01:57:08.596 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'config_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:57:08.589359', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:08.589359', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9854752206779726768, 'uuid_mslong': 3885232349240774114}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0'}, 'oper': 'CREATE', 'request-id': 'req-080b3d52-9fcf-4ddd-853b-6402086b7f46', 'type': 'config_node', 'uuid': '35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0'} 09/01/2025 01:57:08.596 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:08.597 7f2a33db0360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.597 7f2a33db0360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { config_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3885232349240774114, "uuid_lslong": 9854752206779726768}, "enable": true, "created": "2025-09-01T01:57:08.589359", "last_modified": "2025-09-01T01:57:08.589359", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0" } deleted = False >> 09/01/2025 01:57:08.597 7f2a33db0360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:09.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:10.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:10.932 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:10.934 7f2a33db07c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:10.934 7f2a33db07c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:10.934 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:10.934 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:10.934 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:10.934 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:10.935 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:11.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:12.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:13.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:14.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:14.935 7f2a39836040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:14.936 7f2a39836040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:14.936 7f2a39836040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:14.936 7f2a39836040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:14.936 7f2a39836040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:14.936 7f2a39836040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:14.937 7f2a39836040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:14.937 7f2a39836040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:14.937 7f2a39836040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:14.937 7f2a39836040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:14.937 7f2a39836040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:14.937 7f2a39836040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:14.938 7f2a39836040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:14.938 7f2a39836040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:16.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:17.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:18.450 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'config_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:57:18.438111', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.438111', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11481679389149772070, 'uuid_mslong': 9952658429463971103}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8a1ef21c-32ff-451f-9f57-1e9267323926'}, 'oper': 'CREATE', 'request-id': 'req-29c41265-556e-4193-aa1b-64401b965cb7', 'type': 'config_node', 'uuid': '8a1ef21c-32ff-451f-9f57-1e9267323926'} 09/01/2025 01:57:18.450 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:18.451 7f2a39836860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.451 7f2a39836860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { config_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9952658429463971103, "uuid_lslong": 11481679389149772070}, "enable": true, "created": "2025-09-01T01:57:18.438111", "last_modified": "2025-09-01T01:57:18.438111", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "8a1ef21c-32ff-451f-9f57-1e9267323926" } deleted = False >> 09/01/2025 01:57:18.451 7f2a39836860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:18.936 7f2a33db0540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:18.938 7f2a33db0540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:18.938 7f2a33db0540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:18.938 7f2a33db0540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:18.938 7f2a33db0540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:18.938 7f2a33db0540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:18.938 7f2a33db0540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:18.939 7f2a33db0540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:20.898 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:21.899 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.901 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.938 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:22.940 7f2a33db0400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.941 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:22.941 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:22.941 7f2a33db0400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.903 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:24.905 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:25.906 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.907 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.940 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:26.941 7f2a33db0400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:26.941 7f2a33db0400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:26.941 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:26.942 7f2a33db0400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.908 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:28.909 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:29.910 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.912 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.942 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:30.944 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:30.945 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:30.945 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:30.945 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:30.945 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:30.945 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.913 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:32.914 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:33.916 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.920 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.945 7f2a33db0720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:34.948 7f2a33db0720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:34.948 7f2a33db0720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:34.948 7f2a33db0720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:34.948 7f2a33db0720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:34.948 7f2a33db0720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:34.948 7f2a33db0720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:34.949 7f2a33db0720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.921 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:36.922 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:38.802 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:38.949 7f2a33db0cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:38.950 7f2a33db0cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.951 7f2a33db0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:38.951 7f2a33db0cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.951 7f2a33db0cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.951 7f2a33db0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:38.951 7f2a33db0cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.804 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:40.806 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:41.807 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.809 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.950 7f2a39836040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:42.952 7f2a39836040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.953 7f2a39836040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:42.953 7f2a39836040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.953 7f2a39836040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:42.953 7f2a39836040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:42.953 7f2a39836040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.420 7f2a33db0860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.420 7f2a33db0860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:43.421 7f2a33db0860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.470 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:43.460769', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.460769', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11925350455766867329, 'uuid_mslong': 15648137040972304299}}, 'parent_type': 'security-group', 'parent_uuid': '62ca173b-4211-434e-8067-4fcb628e2505', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'}, 'oper': 'CREATE', 'request-id': 'req-e9e6110b-15a8-45e1-b6ff-9bbe5f1606d8', 'type': 'access_control_list', 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'} 09/01/2025 01:57:43.487 7f2a39836ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:egress-access-control-list 09/01/2025 01:57:43.494 7f2a39836ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.249:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "egress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "egress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.519 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:43.512816', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.512816', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11154437056324030764, 'uuid_mslong': 14414019718996774673}}, 'parent_type': 'security-group', 'parent_uuid': '62ca173b-4211-434e-8067-4fcb628e2505', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'}, 'oper': 'CREATE', 'request-id': 'req-a4d2e73b-68f2-4c38-a76b-9509eebec0d9', 'type': 'access_control_list', 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'} 09/01/2025 01:57:43.617 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-09-01T01:57:43.610873', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.610873', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13092489608894395420, 'uuid_mslong': 18272932050593923086}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'fd967ee2-c880-400e-b5b1-ddeeaba4b41c'}, 'oper': 'CREATE', 'request-id': 'req-667bdd9c-a819-416b-a362-7c8709dec24e', 'type': 'route_target', 'uuid': 'fd967ee2-c880-400e-b5b1-ddeeaba4b41c'} 09/01/2025 01:57:43.645 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-d38d9b73-6e23-420e-ad61-26940b0d8417', 'type': 'routing_instance', 'uuid': '26898347-c171-4c1a-8697-64f6fe425338'} 09/01/2025 01:57:43.663 7f2a39836ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000001'] not found in DB: Unknown id: route_target target:64512:8000001 09/01/2025 01:57:43.664 7f2a39836ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000001 url = http://10.0.0.249:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000001"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000001"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.689 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-09-01T01:57:43.671641', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.671641', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13570455496092117547, 'uuid_mslong': 12041940091083179522}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'a71d90b6-3f8c-4602-bc53-f15fd4d16a2b'}, 'oper': 'CREATE', 'request-id': 'req-848a89e9-1bb6-4095-a058-f8b3aa42453b', 'type': 'route_target', 'uuid': 'a71d90b6-3f8c-4602-bc53-f15fd4d16a2b'} 09/01/2025 01:57:43.712 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-7fd48cc7-6227-4e3e-9208-830d1b311a69', 'type': 'routing_instance', 'uuid': 'b402f478-6c09-4cce-978c-bf08921c9092'} 09/01/2025 01:57:43.746 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-09-01T01:57:43.738435', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.738435', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13504741517806881909, 'uuid_mslong': 10871219233836909678}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': '96de544b-843e-446e-bb6a-7adbf8594c75'}, 'oper': 'CREATE', 'request-id': 'req-eafc60a5-2b5d-4bf9-8cf2-e091b0ef8565', 'type': 'route_target', 'uuid': '96de544b-843e-446e-bb6a-7adbf8594c75'} 09/01/2025 01:57:43.755 7f2a39836ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = b092f404-1958-4858-b56e-1b2fb3b6dcd2 object_type = routing_instance identifier_name = default-domain:default-project:default-virtual-network:default-virtual-network url = http://10.0.0.249:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "routing-instance", "uuid": "b092f404-1958-4858-b56e-1b2fb3b6dcd2", "ref-type": "route-target", "ref-uuid": "96de544b-843e-446e-bb6a-7adbf8594c75", "ref-fq-name": ["target:64512:8000002"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.770 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-dd79017c-e717-4d14-ba03-dacf3e729740', 'type': 'routing_instance', 'uuid': 'b092f404-1958-4858-b56e-1b2fb3b6dcd2'} 09/01/2025 01:57:43.810 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:43.814 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-09-01T01:57:43.808184', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.808184', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10211255544152584447, 'uuid_mslong': 14240605882422477187}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'c5a0cb99-a021-4183-8db5-aad0dbf3f0ff'}, 'oper': 'CREATE', 'request-id': 'req-e8279a5b-a4e6-4fee-aa01-864a568ee38c', 'type': 'route_target', 'uuid': 'c5a0cb99-a021-4183-8db5-aad0dbf3f0ff'} 09/01/2025 01:57:43.839 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-94fd4bc4-4c44-489d-8a70-3b0e401279f1', 'type': 'routing_instance', 'uuid': 'fa187b15-223c-4e06-9490-9a6decc1dbc7'} 09/01/2025 01:57:43.932 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:57:43.914671', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.914671', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12434635487439495936, 'uuid_mslong': 17013906886330174589}}, 'parent_type': 'virtual-network', 'parent_uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'}, 'oper': 'CREATE', 'request-id': 'req-7ee2abe6-6195-4e08-b9e2-a1db250d1fde', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 01:57:43.954 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-dacc4d4b-9503-4813-aead-82a4bd87ba77', 'type': 'access_control_list', 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'} 09/01/2025 01:57:43.959 7f2a39836ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c808e290-d047-4f11-9acc-85676ba5d12c object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.249:8082/access-control-list/c808e290-d047-4f11-9acc-85676ba5d12c operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"access-control-list": {"uuid": "c808e290-d047-4f11-9acc-85676ba5d12c", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": 3520976145078303111}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.972 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-28a2300c-ad5b-4055-b6ef-fd6c1340a532', 'type': 'access_control_list', 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'} 09/01/2025 01:57:44.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:45.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.952 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:46.958 7f2a33db07c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:46.958 7f2a33db07c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:46.958 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:46.958 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:46.958 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:46.958 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:46.959 7f2a33db07c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.460 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-b9cee00c-1b45-404e-af54-eadf4c384e92', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:47.560 7f2a39836860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:47.560 7f2a39836860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:47.560 7f2a39836860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:47.664 7f2a39836860 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_linklocal.py remote_ip = 10.0.0.249 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "linklocal_services": {"linklocal_service_entry": [{"linklocal_service_name": "metadata", "linklocal_service_ip": "169.254.169.254", "linklocal_service_port": 80, "ip_fabric_DNS_service_name": "", "ip_fabric_service_port": 8775, "ip_fabric_service_ip": ["10.0.0.249"]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:47.688 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-bf1780e4-3d48-4dbf-97d0-5077fe0b1810', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:47.701 7f2a33db04a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:47.701 7f2a33db04a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:47.702 7f2a33db04a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:47.812 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_linklocal.py remote_ip = 10.0.0.38 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "linklocal_services": {"linklocal_service_entry": [{"linklocal_service_name": "metadata", "linklocal_service_ip": "169.254.169.254", "linklocal_service_port": 80, "ip_fabric_DNS_service_name": "", "ip_fabric_service_port": 8775, "ip_fabric_service_ip": ["10.0.0.249"]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:47.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:47.833 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-f41da2eb-8d35-4dcf-a758-8cf2d89040a0', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:48.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:48.989 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 01:57:48.989 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 01:57:48.990 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 01:57:48.990 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 01:57:48.991 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:48.991 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 01:57:48.991 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 01:57:48.991 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 01:57:49.043 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'] not found in DB: Unknown id: bgp_router default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-1 09/01/2025 01:57:49.050 7f2a380ebe00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-1 url = http://10.0.0.249:8082/bgp-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.249 body = {"bgp-router": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "parent_type": "routing-instance", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.091 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.25', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.25', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T01:57:49.074180', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.074180', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9621979915837513096, 'uuid_mslong': 12350586173020262281}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'}, 'oper': 'CREATE', 'request-id': 'req-c0a3b8a3-b9c0-47b4-aa6d-0410935f0b7e', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:49.092 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.093 7f2a33ea5040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.094 7f2a33ea5040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.074180", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:49.094 7f2a33ea5040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.139 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 09/01/2025 01:57:49.139 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 09/01/2025 01:57:49.139 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 09/01/2025 01:57:49.140 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 09/01/2025 01:57:49.140 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_system_config_back_refs in resource bgp-router: 'global_system_config_back_refs' 09/01/2025 01:57:49.140 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 09/01/2025 01:57:49.140 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:49.140 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 09/01/2025 01:57:49.140 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:49.238 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:49.239 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:49.240 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:49.240 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:49.240 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:49.240 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:49.241 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:49.241 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:49.241 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:49.241 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:49.241 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:49.241 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:49.242 7f2a380ebe00 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:49.296 7f2a380ebe00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/ref-relax-for-delete operation = ref-relax-for-delete useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.249 body = {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ref-uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.351 7f2a380ebe00 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.249 body = {"type": "global-system-config", "uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ref-type": "bgp-router", "ref-uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88", "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "operation": "ADD", "attr": null} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.364 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-bbc43391-f503-4e5f-ae26-aa04ed6e08c3', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:49.645 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 01:57:49.645 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 01:57:49.646 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 01:57:49.646 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 01:57:49.646 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:49.646 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 01:57:49.646 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 01:57:49.646 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 01:57:49.660 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 01:57:49.660 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 01:57:49.660 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 01:57:49.660 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 01:57:49.660 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:49.661 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 01:57:49.661 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 01:57:49.661 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 01:57:49.673 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-elements-inconsistency'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-elements-inconsistency 09/01/2025 01:57:49.683 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-elements-inconsistency"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.697 7f2a33db04a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-elements-inconsistency'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-elements-inconsistency 09/01/2025 01:57:49.699 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'] not found in DB: Unknown id: bgp_router default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-3 09/01/2025 01:57:49.703 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-elements-inconsistency"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.704 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-3 url = http://10.0.0.249:8082/bgp-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.131 body = {"bgp-router": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "parent_type": "routing-instance", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.721 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-01T01:57:49.704740', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.704740', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13465058781904455200, 'uuid_mslong': 269198390335917880}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-7d26a5a0-84f7-42ec-800e-7cdc9b85f184', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.723 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'] not found in DB: Unknown id: bgp_router default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-2 09/01/2025 01:57:49.731 7f2a33db0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-2 url = http://10.0.0.249:8082/bgp-routers operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.38 body = {"bgp-router": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "parent_type": "routing-instance", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.735 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.249:8082/alarms operation = IDs allocation useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-elements-inconsistency", "parent_uuid": "dee9192c-e41b-4855-8333-2236491cfcf8"}} domain = default-domain project = admin user = admin error = alarm:FQ Name: /fq-name-to-uuid/alarm:default-global-system-config:system-defined-conf-elements-inconsistency exists already with ID: 03bc6282-7e14-4f38-badd-7f9f072b4620 at zookeeper >> 09/01/2025 01:57:49.737 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-compute'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-compute 09/01/2025 01:57:49.748 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-compute url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-compute"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_ip_address", "operation": "!=", "operand2": {"uve_attribute": "VrouterAgent.control_ip"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-compute"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.768 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.129', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.129', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'id_perms': {'created': '2025-09-01T01:57:49.750291', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.750291', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11857703135272596087, 'uuid_mslong': 7997815229011673419}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'}, 'oper': 'CREATE', 'request-id': 'req-8f2e7ba5-2d05-4723-9303-eb4f0aa1e337', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:49.768 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.770 7f2a33ea9360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.771 7f2a33ea9360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:49.750291", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:49.771 7f2a33ea9360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.786 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 03bc6282-7e14-4f38-badd-7f9f072b4620 object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.249:8082/alarm/03bc6282-7e14-4f38-badd-7f9f072b4620 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "03bc6282-7e14-4f38-badd-7f9f072b4620", "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.800 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.193', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.193', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T01:57:49.778205', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.778205', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10075118084394099490, 'uuid_mslong': 1863496716102945915}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'}, 'oper': 'CREATE', 'request-id': 'req-e5b882a7-327b-41ce-b42a-1a0c876931dc', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:49.801 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.802 7f2a380ebe00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.802 7f2a380ebe00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:49.778205", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:49.802 7f2a380ebe00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.803 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 03bc6282-7e14-4f38-badd-7f9f072b4620 object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = http://10.0.0.249:8082/alarm/03bc6282-7e14-4f38-badd-7f9f072b4620 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "03bc6282-7e14-4f38-badd-7f9f072b4620", "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:49.822 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 09/01/2025 01:57:49.822 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 09/01/2025 01:57:49.822 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 09/01/2025 01:57:49.822 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 09/01/2025 01:57:49.822 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_system_config_back_refs in resource bgp-router: 'global_system_config_back_refs' 09/01/2025 01:57:49.822 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 09/01/2025 01:57:49.823 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:49.823 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 09/01/2025 01:57:49.823 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 09/01/2025 01:57:49.838 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-02b807bd-14fd-4f1e-b82a-c25261b53d43', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.843 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-01T01:57:49.827148', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.827148', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9281412384941033885, 'uuid_mslong': 18032248181005373295}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4d39c21d-ea49-4fbd-881e-a4e63a6130e5', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:49.852 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-eaedbf1a-4d99-4ee5-a020-a5df73fc9289', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.859 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_system_config_back_refs in resource bgp-router: 'global_system_config_back_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 09/01/2025 01:57:49.860 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 09/01/2025 01:57:49.888 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'oper': 'UPDATE', 'request-id': 'req-3b0d910e-4eff-4ddc-9b7a-42f2a96d0b1a', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:49.895 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-control'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-control 09/01/2025 01:57:49.898 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-3b0d910e-4eff-4ddc-9b7a-42f2a96d0b1a', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:49.899 7f2a380ebe00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.899 7f2a380ebe00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:49.862739", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:49.899 7f2a380ebe00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.907 7f2a380ebe00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.907 7f2a380ebe00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.859846", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:49.907 7f2a380ebe00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.910 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-control url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-control"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operation": "not in", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-control"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.933 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:49.933 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:49.933 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:49.934 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:49.935 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:49.935 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:49.936 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:49.937 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:49.938 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:49.938 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:49.938 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:49.938 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:49.938 7f2a33db07c0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:49.962 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:49.962 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:49.963 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:49.964 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:49.965 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:49.965 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:49.965 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:49.965 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:49.965 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:49.965 7f2a33db0cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:49.998 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/ref-relax-for-delete operation = ref-relax-for-delete useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.131 body = {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ref-uuid": "6efdf293-9873-414b-a48f-062d3df57277"} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.001 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = fa3f6a2e-8282-4f6f-80ce-33639a52499d object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-compute url = http://10.0.0.249:8082/alarm/fa3f6a2e-8282-4f6f-80ce-33639a52499d operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "fa3f6a2e-8282-4f6f-80ce-33639a52499d", "fq_name": ["default-global-system-config", "system-defined-address-mismatch-compute"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_ip_address", "operation": "!=", "operand2": {"uve_attribute": "VrouterAgent.control_ip"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.015 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = fa3f6a2e-8282-4f6f-80ce-33639a52499d object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-compute url = http://10.0.0.249:8082/alarm/fa3f6a2e-8282-4f6f-80ce-33639a52499d operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "fa3f6a2e-8282-4f6f-80ce-33639a52499d", "fq_name": ["default-global-system-config", "system-defined-address-mismatch-compute"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_ip_address", "operation": "!=", "operand2": {"uve_attribute": "VrouterAgent.control_ip"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.027 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'oper': 'UPDATE', 'request-id': 'req-8e0e1c75-cdee-4b81-a97d-47313a4410a5', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:50.034 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-8e0e1c75-cdee-4b81-a97d-47313a4410a5', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:50.037 7f2a33ea9180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.037 7f2a33ea9180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:49.990399", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:50.037 7f2a33ea9180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.040 7f2a33db0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/ref-relax-for-delete operation = ref-relax-for-delete useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.38 body = {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ref-uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.054 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-01T01:57:49.990142', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.990142', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13643264217144023489, 'uuid_mslong': 14422323711782898249}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-d198a7f7-9028-417a-81b9-278fb7ce7f83', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.055 7f2a39836ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 19dc7863-db6c-4c7b-8bd2-02847a6ca322 object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-2 url = http://10.0.0.249:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "bgp-router", "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322", "ref-type": "bgp-router", "ref-uuid": null, "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "operation": "ADD", "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.056 7f2a33ea9180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.056 7f2a33ea9180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.981032", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:50.056 7f2a33ea9180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.064 7f2a33db07c0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.131 body = {"type": "global-system-config", "uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ref-type": "bgp-router", "ref-uuid": "6efdf293-9873-414b-a48f-062d3df57277", "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "operation": "ADD", "attr": null} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.075 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-64ef5a1c-639c-4c3e-b6bf-c4d98fa5bb18', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:50.079 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-bgp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-bgp-connectivity 09/01/2025 01:57:50.083 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-6248bd5b-43c4-48ea-a023-a61144970ae3', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:50.089 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-bgp-connectivity url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-bgp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-bgp-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.093 7f2a33db0cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.38 body = {"type": "global-system-config", "uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ref-type": "bgp-router", "ref-uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322", "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "operation": "ADD", "attr": null} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.098 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-948d4f9f-cccb-41f5-b615-4fdb2635b0d4', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:50.107 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'oper': 'UPDATE', 'request-id': 'req-6f660d61-332b-4f6d-adba-f26f42913c20', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:50.121 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-6f660d61-332b-4f6d-adba-f26f42913c20', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:50.121 7f2a39836860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.121 7f2a39836860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:50.091698", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:50.122 7f2a39836860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.125 7f2a39836860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.125 7f2a39836860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:50.085318", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:50.125 7f2a39836860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.133 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-45aae796-5805-4664-8f68-8e9977d1075b', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:50.162 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.146892', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.146892', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12719190076713101290, 'uuid_mslong': 9355486960643295514}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-ab3956ca-bc75-4fc4-90b9-5fb4002eab77', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.207 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-incorrect'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-incorrect 09/01/2025 01:57:50.212 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-incorrect url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-incorrect"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-incorrect"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.240 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c8266300-cc83-4a49-bd56-9c8537aad9c1 object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-control url = http://10.0.0.249:8082/alarm/c8266300-cc83-4a49-bd56-9c8537aad9c1 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "c8266300-cc83-4a49-bd56-9c8537aad9c1", "fq_name": ["default-global-system-config", "system-defined-address-mismatch-control"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operation": "not in", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.244 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c8266300-cc83-4a49-bd56-9c8537aad9c1 object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-control url = http://10.0.0.249:8082/alarm/c8266300-cc83-4a49-bd56-9c8537aad9c1 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "c8266300-cc83-4a49-bd56-9c8537aad9c1", "fq_name": ["default-global-system-config", "system-defined-address-mismatch-control"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operation": "not in", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.259 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-01T01:57:50.244269', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.244269', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12345975188693128422, 'uuid_mslong': 12789451347087803108}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3ebf1ad0-4d29-4eb6-b68c-83cb5e0c2c57', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.278 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-4644b680-219f-41ac-93f0-2af58d6290f6', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.281 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-a1db2026-671d-43f2-beab-63ca7fd085d9', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.309 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-high'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-high 09/01/2025 01:57:50.318 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-high url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-high"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": "range", "operand2": {"json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-high"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.361 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-01T01:57:50.343229', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.343229', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410398409968651569, 'uuid_mslong': 348915920711273427}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-b3df3838-b993-445d-b342-9bdc65b9c290', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.408 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-critical'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-critical 09/01/2025 01:57:50.416 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-critical url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-critical"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": ">", "operand2": {"json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-critical"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.425 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 81d55dd2-bdf1-491a-b083-a3fc512867ea object_type = alarm identifier_name = default-global-system-config:system-defined-bgp-connectivity url = http://10.0.0.249:8082/alarm/81d55dd2-bdf1-491a-b083-a3fc512867ea operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "81d55dd2-bdf1-491a-b083-a3fc512867ea", "fq_name": ["default-global-system-config", "system-defined-bgp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.430 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 81d55dd2-bdf1-491a-b083-a3fc512867ea object_type = alarm identifier_name = default-global-system-config:system-defined-bgp-connectivity url = http://10.0.0.249:8082/alarm/81d55dd2-bdf1-491a-b083-a3fc512867ea operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "81d55dd2-bdf1-491a-b083-a3fc512867ea", "fq_name": ["default-global-system-config", "system-defined-bgp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.452 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b46a4069-d0ac-41da-baab-6487258bee1f', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.453 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0ea88e73-e57b-474e-b375-5282426f050e', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.463 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-01T01:57:50.437138', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.437138', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10407682553178176690, 'uuid_mslong': 1486187007752490893}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-8cd87a25-7ae6-44da-ac17-b27b16f998eb', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.508 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-node-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-node-status 09/01/2025 01:57:50.516 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-node-status url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-node-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-node-status"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.552 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-01T01:57:50.537284', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.537284', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12650030438488176505, 'uuid_mslong': 7230954819262302855}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-aa98e6eb-63fe-4a1e-abf9-ce5a05ca5b01', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:50.605 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = b17d423d-1f4f-42e4-ab55-b705b7d714e6 object_type = alarm identifier_name = default-global-system-config:system-defined-conf-incorrect url = http://10.0.0.249:8082/alarm/b17d423d-1f4f-42e4-ab55-b705b7d714e6 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "b17d423d-1f4f-42e4-ab55-b705b7d714e6", "fq_name": ["default-global-system-config", "system-defined-conf-incorrect"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig", "operation": "==", "operand2": {"json_value": "null"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.607 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-partial-sysinfo'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-partial-sysinfo 09/01/2025 01:57:50.608 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = b17d423d-1f4f-42e4-ab55-b705b7d714e6 object_type = alarm identifier_name = default-global-system-config:system-defined-conf-incorrect url = http://10.0.0.249:8082/alarm/b17d423d-1f4f-42e4-ab55-b705b7d714e6 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "b17d423d-1f4f-42e4-ab55-b705b7d714e6", "fq_name": ["default-global-system-config", "system-defined-conf-incorrect"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig", "operation": "==", "operand2": {"json_value": "null"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.612 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-partial-sysinfo url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-partial-sysinfo"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.build_info", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "System Info Incomplete.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-partial-sysinfo"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.630 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-a927b894-3265-4f46-a249-ce671ebe63da', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.630 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-939925e0-8ddf-41ea-8f96-f1b4fbe1e932', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.646 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-01T01:57:50.636982', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.636982', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11945006111573889988, 'uuid_mslong': 15521484146632968236}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-772d7b53-1bf5-4336-9bd0-cded7ae36250', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:50.691 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-connectivity 09/01/2025 01:57:50.697 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-connectivity url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_status", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_status.state", "operation": "!=", "operand2": {"json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.725 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.713917', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.713917', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13316427422065746893, 'uuid_mslong': 7023261606246303666}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-17a28f51-9acd-4cc4-9846-b019f65f92ca', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:50.771 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 04d7992e-6679-4bd3-8298-737ed5150d31 object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-high url = http://10.0.0.249:8082/alarm/04d7992e-6679-4bd3-8298-737ed5150d31 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "04d7992e-6679-4bd3-8298-737ed5150d31", "fq_name": ["default-global-system-config", "system-defined-disk-usage-high"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": "range", "operand2": {"json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.772 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-status 09/01/2025 01:57:50.776 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 04d7992e-6679-4bd3-8298-737ed5150d31 object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-high url = http://10.0.0.249:8082/alarm/04d7992e-6679-4bd3-8298-737ed5150d31 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "04d7992e-6679-4bd3-8298-737ed5150d31", "fq_name": ["default-global-system-config", "system-defined-disk-usage-high"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": "range", "operand2": {"json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.778 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-status url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_info", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_info.process_state", "operation": "!=", "operand2": {"json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process Failure.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-status"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.792 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-95b9b3ba-1a0e-4ea3-9245-25030d1a7b8c', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.796 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-ff7124e2-b2c7-4010-879e-a96f7b647965', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.809 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-01T01:57:50.802208', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.802208', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10394145945911723784, 'uuid_mslong': 9299310539720442798}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4f2c310d-2af7-42f0-b0cc-347196801623', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:50.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:50.861 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-connectivity 09/01/2025 01:57:50.867 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-connectivity url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.connected_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.894 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.885227', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:50.885227', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11080958518291472377, 'uuid_mslong': 3112223694598000309}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-c2925086-dcce-45f5-ab84-4caad16fa4c3', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:50.939 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 149fff35-9ae3-4f8d-906f-8423cd373cb2 object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-critical url = http://10.0.0.249:8082/alarm/149fff35-9ae3-4f8d-906f-8423cd373cb2 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "149fff35-9ae3-4f8d-906f-8423cd373cb2", "fq_name": ["default-global-system-config", "system-defined-disk-usage-critical"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": ">", "operand2": {"json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.946 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-tsn-connectivity 09/01/2025 01:57:50.948 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 149fff35-9ae3-4f8d-906f-8423cd373cb2 object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-critical url = http://10.0.0.249:8082/alarm/149fff35-9ae3-4f8d-906f-8423cd373cb2 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "149fff35-9ae3-4f8d-906f-8423cd373cb2", "fq_name": ["default-global-system-config", "system-defined-disk-usage-critical"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": ">", "operand2": {"json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.953 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-tsn-connectivity url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-tsn-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}, {"operand1": "ProuterData.gateway_mode", "operation": "!=", "operand2": {"json_value": "\"SERVER\""}}]}, {"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "0"}}, {"operand1": "ProuterData.gateway_mode", "operation": "==", "operand2": {"json_value": "\"SERVER\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-tsn-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:50.960 7f2a33ea99a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:50.962 7f2a33ea99a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:50.962 7f2a33ea99a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:50.962 7f2a33ea99a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:50.962 7f2a33ea99a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:50.962 7f2a33ea99a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:50.962 7f2a33ea99a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:50.963 7f2a33ea99a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:50.963 7f2a33ea99a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.964 7f2a33ea99a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:50.964 7f2a33ea99a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.964 7f2a33ea99a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.964 7f2a33ea99a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:50.964 7f2a33ea99a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.968 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-6d2502b7-dea2-4907-81a1-e4b8a763d3ee', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.975 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-0b329c1d-2db3-448f-97e9-07c801904437', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.991 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.981229', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:50.981229', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10109986246743945171, 'uuid_mslong': 10619123826018765353}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-089282b9-7dc7-43b2-a6ef-7aaed496978e', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:51.035 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-storage-cluster-state'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-storage-cluster-state 09/01/2025 01:57:51.040 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-storage-cluster-state url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-storage-cluster-state"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["storage-cluster"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "StorageCluster.info_stats.status", "operation": "!=", "operand2": {"json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-storage-cluster-state"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.074 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-01T01:57:51.062109', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.062109', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11650388398251038770, 'uuid_mslong': 10237949505899020380}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-ddf9ec54-1868-42ef-b16e-0e94d2b51aad', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:51.116 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 64598318-cd07-4a87-af8d-efa93bf77779 object_type = alarm identifier_name = default-global-system-config:system-defined-node-status url = http://10.0.0.249:8082/alarm/64598318-cd07-4a87-af8d-efa93bf77779 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "64598318-cd07-4a87-af8d-efa93bf77779", "fq_name": ["default-global-system-config", "system-defined-node-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus", "operation": "==", "operand2": {"json_value": "null"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.119 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 64598318-cd07-4a87-af8d-efa93bf77779 object_type = alarm identifier_name = default-global-system-config:system-defined-node-status url = http://10.0.0.249:8082/alarm/64598318-cd07-4a87-af8d-efa93bf77779 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "64598318-cd07-4a87-af8d-efa93bf77779", "fq_name": ["default-global-system-config", "system-defined-node-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus", "operation": "==", "operand2": {"json_value": "null"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.120 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-interface'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-interface 09/01/2025 01:57:51.125 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-interface url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-interface"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.down_interface_count", "operation": ">=", "operand2": {"json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-interface"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.136 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-40c61117-53d3-4682-b6aa-ebab239dc2f6', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:51.141 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-372e3cc2-6865-4786-b23f-a0ea8df70b4e', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:51.157 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-01T01:57:51.143961', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.143961', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11278860563117070971, 'uuid_mslong': 11803345335316596501}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-1a9eb67e-f428-4a37-96a9-72466ae74d61', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:51.203 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-connectivity 09/01/2025 01:57:51.209 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-connectivity url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-connectivity"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.238 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:51.226178', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.226178', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11143410889604691103, 'uuid_mslong': 14282259945798913346}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-32967334-05f5-451f-82ec-3c34074acc51', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:51.281 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = d7676398-52ea-482c-a5c5-2fc5683b53c4 object_type = alarm identifier_name = default-global-system-config:system-defined-partial-sysinfo url = http://10.0.0.249:8082/alarm/d7676398-52ea-482c-a5c5-2fc5683b53c4 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "d7676398-52ea-482c-a5c5-2fc5683b53c4", "fq_name": ["default-global-system-config", "system-defined-partial-sysinfo"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.build_info", "operation": "==", "operand2": {"json_value": "null"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.282 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-close-reason'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-close-reason 09/01/2025 01:57:51.289 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-close-reason url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-close-reason"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["xmpp-peer"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "XmppPeerInfoData.close_reason", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "XmppPeerInfoData.state_info.state", "operation": "!=", "operand2": {"json_value": "\"Established\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-close-reason"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.296 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = d7676398-52ea-482c-a5c5-2fc5683b53c4 object_type = alarm identifier_name = default-global-system-config:system-defined-partial-sysinfo url = http://10.0.0.249:8082/alarm/d7676398-52ea-482c-a5c5-2fc5683b53c4 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "d7676398-52ea-482c-a5c5-2fc5683b53c4", "fq_name": ["default-global-system-config", "system-defined-partial-sysinfo"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.build_info", "operation": "==", "operand2": {"json_value": "null"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.302 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-b459fe8b-724f-444f-8092-ca948a7c4bb7', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:51.315 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-c1fbc6a9-f5aa-4b42-83a9-874da7552f7a', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:51.330 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-01T01:57:51.312606', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-01T01:57:51.312606', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13487801257560256231, 'uuid_mslong': 10235680329170243091}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-3c23439e-0ccb-4d2b-b98a-24cb37aaab0c', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:51.376 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-core-files'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-core-files 09/01/2025 01:57:51.382 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-core-files url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-core-files"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "null"}, "operation": "!="}, {"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "0"}, "operation": "size!="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-core-files"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.408 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-01T01:57:51.398898', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.398898', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10710412090359268505, 'uuid_mslong': 3024502048848363658}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-da6c1f03-ff69-4dc6-a1ce-b0a85f4b848d', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:51.447 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 6177a338-54f3-47b2-b8cd-742fd97fe3cd object_type = alarm identifier_name = default-global-system-config:system-defined-process-connectivity url = http://10.0.0.249:8082/alarm/6177a338-54f3-47b2-b8cd-742fd97fe3cd operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "6177a338-54f3-47b2-b8cd-742fd97fe3cd", "fq_name": ["default-global-system-config", "system-defined-process-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_status", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_status.state", "operation": "!=", "operand2": {"json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.454 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-pending-cassandra-compaction-tasks 09/01/2025 01:57:51.461 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 6177a338-54f3-47b2-b8cd-742fd97fe3cd object_type = alarm identifier_name = default-global-system-config:system-defined-process-connectivity url = http://10.0.0.249:8082/alarm/6177a338-54f3-47b2-b8cd-742fd97fe3cd operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "6177a338-54f3-47b2-b8cd-742fd97fe3cd", "fq_name": ["default-global-system-config", "system-defined-process-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_status", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_status.state", "operation": "!=", "operand2": {"json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.470 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["database-nodeconfig-database-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"json_value": "300"}, "operation": ">="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-pending-cassandra-compaction-tasks"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.475 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-86a74659-a45f-42ec-840b-7a8d9d5f0e2f', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:51.485 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-ff070e25-e45d-4fe3-be1d-003d05e1b568', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:51.502 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-01T01:57:51.495240', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.495240', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12799166691106454208, 'uuid_mslong': 15027170893856260812}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-94e1d593-b6f7-40fa-a257-80aad936d295', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:51.549 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-package-version-mismatch'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-package-version-mismatch 09/01/2025 01:57:51.555 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-package-version-mismatch url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-package-version-mismatch"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.running_package_version", "operation": "!=", "operand2": {"uve_attribute": "NodeStatus.installed_package_version"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-package-version-mismatch"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.586 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-01T01:57:51.577756', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.577756', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10722918137435541080, 'uuid_mslong': 17809913453040846757}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-220998bd-94b4-4899-848b-14a09e1c3183', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:51.624 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 810dc9aa-8015-43ae-903f-6caad1a28708 object_type = alarm identifier_name = default-global-system-config:system-defined-process-status url = http://10.0.0.249:8082/alarm/810dc9aa-8015-43ae-903f-6caad1a28708 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "810dc9aa-8015-43ae-903f-6caad1a28708", "fq_name": ["default-global-system-config", "system-defined-process-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_info", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_info.process_state", "operation": "!=", "operand2": {"json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.635 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-limit-exceeded 09/01/2025 01:57:51.640 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 810dc9aa-8015-43ae-903f-6caad1a28708 object_type = alarm identifier_name = default-global-system-config:system-defined-process-status url = http://10.0.0.249:8082/alarm/810dc9aa-8015-43ae-903f-6caad1a28708 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "810dc9aa-8015-43ae-903f-6caad1a28708", "fq_name": ["default-global-system-config", "system-defined-process-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_info", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_info.process_state", "operation": "!=", "operand2": {"json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.642 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-limit-exceeded url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-limit-exceeded"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.648 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-1b0fb202-a15f-4578-a079-413e846b84b2', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:51.660 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-5ce20252-7ee3-433a-afd4-97a83e0e1942', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:51.676 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:51.666562', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.666562', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10608329397487415044, 'uuid_mslong': 12641009152992233842}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-043efcd0-fec7-4d8c-b54c-f5c2f0bffb7e', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:51.726 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-table-limit-exceeded 09/01/2025 01:57:51.732 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-table-limit-exceeded url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_table_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-table-limit-exceeded"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.765 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:51.751034', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.751034', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12665274423476574180, 'uuid_mslong': 14559706905692358710}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-2b1fdcaf-0532-49a7-89d0-ff8f0b532112', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:51.792 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 2b30d6f6-0211-4eb5-99c7-79101d9a1ff9 object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-connectivity url = http://10.0.0.249:8082/alarm/2b30d6f6-0211-4eb5-99c7-79101d9a1ff9 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "2b30d6f6-0211-4eb5-99c7-79101d9a1ff9", "fq_name": ["default-global-system-config", "system-defined-prouter-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.connected_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.804 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 2b30d6f6-0211-4eb5-99c7-79101d9a1ff9 object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-connectivity url = http://10.0.0.249:8082/alarm/2b30d6f6-0211-4eb5-99c7-79101d9a1ff9 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "2b30d6f6-0211-4eb5-99c7-79101d9a1ff9", "fq_name": ["default-global-system-config", "system-defined-prouter-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.connected_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.812 7f2a33db0400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down 09/01/2025 01:57:51.820 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-cceef24b-02aa-4ba9-8e77-159d86d7f996', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:51.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:51.823 7f2a33db0400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down url = http://10.0.0.249:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.131 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.l3mh_phy_interface_down", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-l3mh-physical-interface-link-down"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.831 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-9e20dffb-8801-43ec-8834-42aff3338aa0', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:51.861 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-01T01:57:51.847772', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.847772', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11156140797610135132, 'uuid_mslong': 437379043040970971}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-6e9b4c17-ad10-4551-8af6-958c601596a7', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:51.975 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 935eb4db-73bc-4a29-8c4d-e2ed2941efd3 object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-tsn-connectivity url = http://10.0.0.249:8082/alarm/935eb4db-73bc-4a29-8c4d-e2ed2941efd3 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "935eb4db-73bc-4a29-8c4d-e2ed2941efd3", "fq_name": ["default-global-system-config", "system-defined-prouter-tsn-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}, {"operand1": "ProuterData.gateway_mode", "operation": "!=", "operand2": {"json_value": "\"SERVER\""}}]}, {"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "0"}}, {"operand1": "ProuterData.gateway_mode", "operation": "==", "operand2": {"json_value": "\"SERVER\""}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:51.985 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 935eb4db-73bc-4a29-8c4d-e2ed2941efd3 object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-tsn-connectivity url = http://10.0.0.249:8082/alarm/935eb4db-73bc-4a29-8c4d-e2ed2941efd3 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "935eb4db-73bc-4a29-8c4d-e2ed2941efd3", "fq_name": ["default-global-system-config", "system-defined-prouter-tsn-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}, {"operand1": "ProuterData.gateway_mode", "operation": "!=", "operand2": {"json_value": "\"SERVER\""}}]}, {"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "0"}}, {"operand1": "ProuterData.gateway_mode", "operation": "==", "operand2": {"json_value": "\"SERVER\""}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.006 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0ce38428-da3d-46d3-af77-6621d179b916', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:52.038 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-4a3dbbdb-8ea8-4bb0-b802-e55592a31842', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:52.156 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 8e1480d5-277f-405c-a1ae-7e859515d832 object_type = alarm identifier_name = default-global-system-config:system-defined-storage-cluster-state url = http://10.0.0.249:8082/alarm/8e1480d5-277f-405c-a1ae-7e859515d832 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "8e1480d5-277f-405c-a1ae-7e859515d832", "fq_name": ["default-global-system-config", "system-defined-storage-cluster-state"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["storage-cluster"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "StorageCluster.info_stats.status", "operation": "!=", "operand2": {"json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.190 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 8e1480d5-277f-405c-a1ae-7e859515d832 object_type = alarm identifier_name = default-global-system-config:system-defined-storage-cluster-state url = http://10.0.0.249:8082/alarm/8e1480d5-277f-405c-a1ae-7e859515d832 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "8e1480d5-277f-405c-a1ae-7e859515d832", "fq_name": ["default-global-system-config", "system-defined-storage-cluster-state"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["storage-cluster"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "StorageCluster.info_stats.status", "operation": "!=", "operand2": {"json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.202 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-903126b7-3e5d-4c0d-a96c-0112009b6b6b', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:52.228 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-574ad662-aa66-4387-b6f3-ae5ecedfb9c0', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:52.363 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = a3cde80b-b0cf-4715-9c86-8fec97c15a7b object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-interface url = http://10.0.0.249:8082/alarm/a3cde80b-b0cf-4715-9c86-8fec97c15a7b operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "a3cde80b-b0cf-4715-9c86-8fec97c15a7b", "fq_name": ["default-global-system-config", "system-defined-vrouter-interface"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.down_interface_count", "operation": ">=", "operand2": {"json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.382 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = a3cde80b-b0cf-4715-9c86-8fec97c15a7b object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-interface url = http://10.0.0.249:8082/alarm/a3cde80b-b0cf-4715-9c86-8fec97c15a7b operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "a3cde80b-b0cf-4715-9c86-8fec97c15a7b", "fq_name": ["default-global-system-config", "system-defined-vrouter-interface"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.down_interface_count", "operation": ">=", "operand2": {"json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.415 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-65485163-b46b-4b7d-b4ea-ac3e45e48b1b', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:52.418 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-4053dbbf-3e1e-47e0-bc3a-9121239cf4a1', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:52.559 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c634c7c0-0322-4d42-9aa5-5929ed1b749f object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-connectivity url = http://10.0.0.249:8082/alarm/c634c7c0-0322-4d42-9aa5-5929ed1b749f operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "c634c7c0-0322-4d42-9aa5-5929ed1b749f", "fq_name": ["default-global-system-config", "system-defined-xmpp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.583 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = c634c7c0-0322-4d42-9aa5-5929ed1b749f object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-connectivity url = http://10.0.0.249:8082/alarm/c634c7c0-0322-4d42-9aa5-5929ed1b749f operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "c634c7c0-0322-4d42-9aa5-5929ed1b749f", "fq_name": ["default-global-system-config", "system-defined-xmpp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.610 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b5885dd0-54b6-4de9-b985-e5729ac70731', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:52.643 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-c98a9e53-e0a4-47ea-a8a2-f875f5abdca0', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:52.778 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 8e0c7107-46a5-4e13-bb2e-4bc845f22ee7 object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-close-reason url = http://10.0.0.249:8082/alarm/8e0c7107-46a5-4e13-bb2e-4bc845f22ee7 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "8e0c7107-46a5-4e13-bb2e-4bc845f22ee7", "fq_name": ["default-global-system-config", "system-defined-xmpp-close-reason"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["xmpp-peer"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "XmppPeerInfoData.close_reason", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "XmppPeerInfoData.state_info.state", "operation": "!=", "operand2": {"json_value": "\"Established\""}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.780 7f2a33ea9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_encap.py remote_ip = 10.0.0.131 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "encapsulation_priorities": {"encapsulation": ["MPLSoUDP", "MPLSoGRE", "VXLAN"]}, "vxlan_network_identifier_mode": "automatic"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.790 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 8e0c7107-46a5-4e13-bb2e-4bc845f22ee7 object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-close-reason url = http://10.0.0.249:8082/alarm/8e0c7107-46a5-4e13-bb2e-4bc845f22ee7 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "8e0c7107-46a5-4e13-bb2e-4bc845f22ee7", "fq_name": ["default-global-system-config", "system-defined-xmpp-close-reason"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["xmpp-peer"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "XmppPeerInfoData.close_reason", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "XmppPeerInfoData.state_info.state", "operation": "!=", "operand2": {"json_value": "\"Established\""}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.798 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-be77af09-c6a1-4809-9837-02e4960fbc0f', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:52.808 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-8a8b9aea-1dd9-49ef-a3fa-731e869e0879', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:52.816 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-f824c559-dd9e-47f6-b43c-74bdb695c193', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:52.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:52.965 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 29f93096-c40c-408a-94a3-070ccbe9c499 object_type = alarm identifier_name = default-global-system-config:system-defined-core-files url = http://10.0.0.249:8082/alarm/29f93096-c40c-408a-94a3-070ccbe9c499 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "29f93096-c40c-408a-94a3-070ccbe9c499", "fq_name": ["default-global-system-config", "system-defined-core-files"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "null"}, "operation": "!="}, {"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "0"}, "operation": "size!="}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.966 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 29f93096-c40c-408a-94a3-070ccbe9c499 object_type = alarm identifier_name = default-global-system-config:system-defined-core-files url = http://10.0.0.249:8082/alarm/29f93096-c40c-408a-94a3-070ccbe9c499 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "29f93096-c40c-408a-94a3-070ccbe9c499", "fq_name": ["default-global-system-config", "system-defined-core-files"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "null"}, "operation": "!="}, {"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "0"}, "operation": "size!="}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:52.982 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-5978a90a-a2bb-4e72-b492-04055837bab6', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:52.986 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-cc861575-b1f8-4b01-8968-321ce8a165f7', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:53.126 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = d08b3c4f-8364-42cc-b19f-c64aec697ac0 object_type = alarm identifier_name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks url = http://10.0.0.249:8082/alarm/d08b3c4f-8364-42cc-b19f-c64aec697ac0 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "d08b3c4f-8364-42cc-b19f-c64aec697ac0", "fq_name": ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["database-nodeconfig-database-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"json_value": "300"}, "operation": ">="}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.131 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = d08b3c4f-8364-42cc-b19f-c64aec697ac0 object_type = alarm identifier_name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks url = http://10.0.0.249:8082/alarm/d08b3c4f-8364-42cc-b19f-c64aec697ac0 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "d08b3c4f-8364-42cc-b19f-c64aec697ac0", "fq_name": ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["database-nodeconfig-database-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"json_value": "300"}, "operation": ">="}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.148 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-8fd3efd9-9811-4d74-9c37-1d6a2f9d9316', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:53.164 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-d9273f8f-b94b-4a39-bdac-d445fbe24224', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:53.297 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = f72985ec-b0f3-47a5-94cf-753bc9704e58 object_type = alarm identifier_name = default-global-system-config:system-defined-package-version-mismatch url = http://10.0.0.249:8082/alarm/f72985ec-b0f3-47a5-94cf-753bc9704e58 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "f72985ec-b0f3-47a5-94cf-753bc9704e58", "fq_name": ["default-global-system-config", "system-defined-package-version-mismatch"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.running_package_version", "operation": "!=", "operand2": {"uve_attribute": "NodeStatus.installed_package_version"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.316 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = f72985ec-b0f3-47a5-94cf-753bc9704e58 object_type = alarm identifier_name = default-global-system-config:system-defined-package-version-mismatch url = http://10.0.0.249:8082/alarm/f72985ec-b0f3-47a5-94cf-753bc9704e58 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "f72985ec-b0f3-47a5-94cf-753bc9704e58", "fq_name": ["default-global-system-config", "system-defined-package-version-mismatch"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.running_package_version", "operation": "!=", "operand2": {"uve_attribute": "NodeStatus.installed_package_version"}}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.336 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-ea9f552e-98eb-4ef4-80bc-5642d6297bf3', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:53.345 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-e3763da9-84a2-42ee-81cf-a733fdda7b1f', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:53.454 7f2a33ea9cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:53.455 7f2a33ea9cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:53.455 7f2a33ea9cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:53.493 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = af6de2d9-8690-4972-9338-5b61b5922f04 object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-limit-exceeded url = http://10.0.0.249:8082/alarm/af6de2d9-8690-4972-9338-5b61b5922f04 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "af6de2d9-8690-4972-9338-5b61b5922f04", "fq_name": ["default-global-system-config", "system-defined-vrouter-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.495 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = af6de2d9-8690-4972-9338-5b61b5922f04 object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-limit-exceeded url = http://10.0.0.249:8082/alarm/af6de2d9-8690-4972-9338-5b61b5922f04 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "af6de2d9-8690-4972-9338-5b61b5922f04", "fq_name": ["default-global-system-config", "system-defined-vrouter-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.514 7f2a33ea9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_global_vrouter_config.py remote_ip = 10.0.0.131 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "port_translation_pools": {"port_translation_pool": []}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.529 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-00c0f81c-27b8-4bf7-b45d-8c60f682ff4c', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:53.532 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-61c5b9c6-20c7-4119-a5a5-c3f28a485fa3', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:53.535 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-fc63db98-24bf-4e4b-a4ef-4f72524f73db', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:53.669 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = ca0e784a-4398-4836-afc4-17fc122957e4 object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-table-limit-exceeded url = http://10.0.0.249:8082/alarm/ca0e784a-4398-4836-afc4-17fc122957e4 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "ca0e784a-4398-4836-afc4-17fc122957e4", "fq_name": ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_table_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.674 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = ca0e784a-4398-4836-afc4-17fc122957e4 object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-table-limit-exceeded url = http://10.0.0.249:8082/alarm/ca0e784a-4398-4836-afc4-17fc122957e4 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "ca0e784a-4398-4836-afc4-17fc122957e4", "fq_name": ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_table_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.688 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-3c1aef7d-30c2-406e-983f-8730edafccde', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:53.696 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-e4842649-b912-4285-b75d-22e7fdce5d46', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:53.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:53.830 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 0611e1ec-1ca8-48db-9ad2-92f299cb8e5c object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down url = http://10.0.0.249:8082/alarm/0611e1ec-1ca8-48db-9ad2-92f299cb8e5c operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.249 body = {"alarm": {"uuid": "0611e1ec-1ca8-48db-9ad2-92f299cb8e5c", "fq_name": ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.l3mh_phy_interface_down", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.858 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 0611e1ec-1ca8-48db-9ad2-92f299cb8e5c object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down url = http://10.0.0.249:8082/alarm/0611e1ec-1ca8-48db-9ad2-92f299cb8e5c operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.38 body = {"alarm": {"uuid": "0611e1ec-1ca8-48db-9ad2-92f299cb8e5c", "fq_name": ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.l3mh_phy_interface_down", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:53.865 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-257ec96c-0a05-47d5-8ed5-bceba5186da5', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:53.879 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-e087c260-1b7a-4195-aa63-9353258567c6', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:54.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:54.845 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:54.846 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:54.847 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:54.848 7f2a33d23cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:54.855 7f2a33d23cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/global-system-config/dee9192c-e41b-4855-8333-2236491cfcf8 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.131 body = {"global-system-config": {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "autonomous_system": 64512, "ibgp_auto_mesh": true}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:54.879 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-02593c6e-2575-40b8-a56f-63f0b1fa86fb', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:54.908 7f2a33db04a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_encap.py remote_ip = 10.0.0.249 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "encapsulation_priorities": {"encapsulation": ["MPLSoUDP", "MPLSoGRE", "VXLAN"]}, "vxlan_network_identifier_mode": "automatic"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:54.922 7f2a33db0ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_encap.py remote_ip = 10.0.0.38 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "encapsulation_priorities": {"encapsulation": ["MPLSoUDP", "MPLSoGRE", "VXLAN"]}, "vxlan_network_identifier_mode": "automatic"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:54.951 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-9a1cd17d-a4e0-4fed-a459-be82136524dd', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:54.954 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-4438a8df-7099-48db-8441-b15938d14eed', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:54.962 7f2a33d23cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:54.963 7f2a33d23cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:54.963 7f2a33d23cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:54.963 7f2a33d23cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:54.964 7f2a33d23cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:56.099 7f2a33d23d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:56.099 7f2a33d23d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:56.099 7f2a33d23d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:56.154 7f2a33db0ea0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Object UUID cache manager: set cache already there, before last modified: 1756691874947751 and meta row: 0 after last modified: 1756691874947751 and meta row: 0 09/01/2025 01:57:56.155 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:56.156 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:56.156 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:56.212 7f2a33d23d60 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_global_vrouter_config.py remote_ip = 10.0.0.38 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "port_translation_pools": {"port_translation_pool": []}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:56.230 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-425b5a83-c87b-46df-8e2c-db3c3f271101', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:56.261 7f2a33db0ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.249:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_global_vrouter_config.py remote_ip = 10.0.0.249 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "port_translation_pools": {"port_translation_pool": []}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:56.273 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-6c0bdafe-bb88-447a-b0d8-8cbade3c0907', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:56.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:57.419 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:57.419 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:57.419 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:57.420 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:57.421 7f2a33db0ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:57.434 7f2a33db0ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/global-system-config/dee9192c-e41b-4855-8333-2236491cfcf8 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.38 body = {"global-system-config": {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "autonomous_system": 64512, "ibgp_auto_mesh": true}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:57.436 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:57:57.436 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:57:57.436 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:57:57.436 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:57:57.436 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:57:57.437 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:57:57.438 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:57:57.439 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:57:57.440 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:57:57.440 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:57:57.440 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:57:57.440 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:57:57.440 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:57:57.440 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:57:57.441 7f2a380c64a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:57:57.471 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-f00a55d7-ce9e-43e2-9395-3dd8285232ff', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:57.498 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/global-system-config/dee9192c-e41b-4855-8333-2236491cfcf8 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.249 body = {"global-system-config": {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "autonomous_system": 64512, "ibgp_auto_mesh": true}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:57.533 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-3b996c4a-3334-4609-ac9b-97b4439f5e60', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:57.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:58.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:58.964 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:58.965 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:58.965 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:58.965 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:58.965 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:58.965 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:58.965 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:58.966 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:00.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:01.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:02.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:02.965 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:02.966 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:02.966 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:02.967 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:02.968 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:04.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:05.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:06.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:06.967 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:06.968 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:06.968 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:06.968 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:06.969 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:08.803 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.804 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:10.805 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:10.969 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:10.970 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.971 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.972 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:10.972 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.808 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:12.809 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.811 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:14.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:14.971 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:14.972 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:14.972 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:14.972 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:14.972 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:14.972 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:14.972 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:14.973 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:16.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:17.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:18.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:18.972 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:18.982 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:18.982 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:18.982 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:18.982 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:18.982 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:18.982 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:18.983 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:20.166 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:20.167 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:20.167 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:20.167 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:20.167 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:20.167 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:20.167 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:20.168 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:20.169 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:20.170 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:20.171 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:20.171 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:20.185 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:20.185 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:20.185 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:20.186 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:20.187 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:20.188 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:20.189 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:20.189 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:20.189 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:20.189 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:20.189 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:20.189 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:20.213 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:20.214 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:20.215 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:20.233 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'] not found in DB: Unknown id: database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. 09/01/2025 01:58:20.241 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. url = http://10.0.0.249:8082/database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_database_node.py remote_ip = 10.0.0.38 body = {"database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."], "parent_type": "global-system-config", "database_node_ip_address": "10.0.0.38", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:20.243 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'] not found in DB: Unknown id: database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. 09/01/2025 01:58:20.247 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. url = http://10.0.0.249:8082/database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_database_node.py remote_ip = 10.0.0.131 body = {"database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."], "parent_type": "global-system-config", "database_node_ip_address": "10.0.0.131", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:20.278 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'database_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:58:20.262077', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.262077', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11796017848418923458, 'uuid_mslong': 4531286052305127849}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3ee25ca8-2354-49a9-a3b3-dfbc332d73c2'}, 'oper': 'CREATE', 'request-id': 'req-89536832-5ade-4091-a3e7-dc5b51518fac', 'type': 'database_node', 'uuid': '3ee25ca8-2354-49a9-a3b3-dfbc332d73c2'} 09/01/2025 01:58:20.278 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.279 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.279 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { database_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4531286052305127849, "uuid_lslong": 11796017848418923458}, "enable": true, "created": "2025-09-01T01:58:20.262077", "last_modified": "2025-09-01T01:58:20.262077", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "3ee25ca8-2354-49a9-a3b3-dfbc332d73c2" } deleted = False >> 09/01/2025 01:58:20.280 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.290 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'database_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:58:20.280218', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.280218', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10071480984782399945, 'uuid_mslong': 378361382186796020}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '054035ac-25f8-43f4-8bc5-16983c0e69c9'}, 'oper': 'CREATE', 'request-id': 'req-1e468cdd-34f7-445d-b18b-bd3cca7149c2', 'type': 'database_node', 'uuid': '054035ac-25f8-43f4-8bc5-16983c0e69c9'} 09/01/2025 01:58:20.290 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.291 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.291 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { database_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 378361382186796020, "uuid_lslong": 10071480984782399945}, "enable": true, "created": "2025-09-01T01:58:20.280218", "last_modified": "2025-09-01T01:58:20.280218", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "054035ac-25f8-43f4-8bc5-16983c0e69c9" } deleted = False >> 09/01/2025 01:58:20.291 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.329 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'] not found in DB: Unknown id: database_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. 09/01/2025 01:58:20.348 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. url = http://10.0.0.249:8082/database-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_database_node.py remote_ip = 10.0.0.249 body = {"database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."], "parent_type": "global-system-config", "database_node_ip_address": "10.0.0.249", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:20.408 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:58:20.395054', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.395054', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12835641244198905252, 'uuid_mslong': 4848350493246179075}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4348cd16-0087-4703-b221-5bb498d425a4'}, 'oper': 'CREATE', 'request-id': 'req-67f78842-b75a-435a-8bb4-1eea81bd8c1f', 'type': 'database_node', 'uuid': '4348cd16-0087-4703-b221-5bb498d425a4'} 09/01/2025 01:58:20.408 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.409 7f2a380c62c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.409 7f2a380c62c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { database_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4848350493246179075, "uuid_lslong": 12835641244198905252}, "enable": true, "created": "2025-09-01T01:58:20.395054", "last_modified": "2025-09-01T01:58:20.395054", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "4348cd16-0087-4703-b221-5bb498d425a4" } deleted = False >> 09/01/2025 01:58:20.409 7f2a380c62c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:21.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:22.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:22.983 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:22.984 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:22.985 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:22.985 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:22.985 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:22.985 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:22.985 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:22.986 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:22.987 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.987 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:22.987 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.987 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.987 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:22.987 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:24.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:25.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:26.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:26.985 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:26.986 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.987 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.988 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:26.988 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:28.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:29.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.988 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:30.989 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:30.989 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:30.989 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:30.990 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:30.990 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:30.990 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:30.990 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:30.990 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:30.992 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:30.992 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:30.992 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:30.992 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:30.992 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:32.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:33.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.991 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:34.992 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:34.993 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:34.994 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:34.994 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:34.994 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:34.994 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:34.994 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:36.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:38.805 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:38.993 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:38.994 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:38.994 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:38.995 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.807 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:40.808 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.810 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:42.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:42.995 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:43.002 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:43.002 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:43.002 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:43.003 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:43.003 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:43.003 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:43.003 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:43.003 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.003 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:43.004 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.004 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.004 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:43.004 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:44.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:45.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:46.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:47.002 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:47.004 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:47.005 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.005 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:47.005 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.005 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.005 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:47.005 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:48.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:49.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:50.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:51.005 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:51.012 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.013 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:51.013 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.013 7f2a38273cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.013 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:51.013 7f2a38273cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:52.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:53.852 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:53.853 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:53.854 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:53.865 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:53.866 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:53.867 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:53.868 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:53.890 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:58:53.890 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:58:53.890 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:58:53.890 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:58:53.890 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:58:53.891 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:58:53.892 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:58:53.893 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:58:53.893 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:58:53.893 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:58:53.916 7f2a38273cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'] not found in DB: Unknown id: analytics_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. 09/01/2025 01:58:53.926 7f2a38273cc0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. url = http://10.0.0.249:8082/analytics-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_analytics_node.py remote_ip = 10.0.0.38 body = {"analytics-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."], "parent_type": "global-system-config", "analytics_node_ip_address": "10.0.0.38", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:53.939 7f2a380c6720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'] not found in DB: Unknown id: analytics_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. 09/01/2025 01:58:53.945 7f2a380c6720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. url = http://10.0.0.249:8082/analytics-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_analytics_node.py remote_ip = 10.0.0.131 body = {"analytics-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."], "parent_type": "global-system-config", "analytics_node_ip_address": "10.0.0.131", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:53.968 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:58:53.957876', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:53.957876', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13676038583231394089, 'uuid_mslong': 7839128237629393148}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6cca2d99-40a2-48fc-bdcb-0ca1f80b3129'}, 'oper': 'CREATE', 'request-id': 'req-f2c45455-00f1-4b72-b900-1a1e9bad38ff', 'type': 'analytics_node', 'uuid': '6cca2d99-40a2-48fc-bdcb-0ca1f80b3129'} 09/01/2025 01:58:53.968 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:53.968 7f2a38d07ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.969 7f2a38d07ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7839128237629393148, "uuid_lslong": 13676038583231394089}, "enable": true, "created": "2025-09-01T01:58:53.957876", "last_modified": "2025-09-01T01:58:53.957876", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6cca2d99-40a2-48fc-bdcb-0ca1f80b3129" } deleted = False >> 09/01/2025 01:58:53.969 7f2a38d07ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.986 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:58:53.973832', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:53.973832', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10557226880738800812, 'uuid_mslong': 4644487576518083042}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '407488d7-e213-41e2-9282-cde9d7a434ac'}, 'oper': 'CREATE', 'request-id': 'req-7052381b-b338-4995-aa45-2d649573240e', 'type': 'analytics_node', 'uuid': '407488d7-e213-41e2-9282-cde9d7a434ac'} 09/01/2025 01:58:53.986 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:53.988 7f2a38d07ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.988 7f2a38d07ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4644487576518083042, "uuid_lslong": 10557226880738800812}, "enable": true, "created": "2025-09-01T01:58:53.973832", "last_modified": "2025-09-01T01:58:53.973832", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "407488d7-e213-41e2-9282-cde9d7a434ac" } deleted = False >> 09/01/2025 01:58:53.988 7f2a38d07ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.000 7f2a380c6ea0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'] not found in DB: Unknown id: analytics_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. 09/01/2025 01:58:54.021 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. url = http://10.0.0.249:8082/analytics-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_analytics_node.py remote_ip = 10.0.0.249 body = {"analytics-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."], "parent_type": "global-system-config", "analytics_node_ip_address": "10.0.0.249", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:58:54.082 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:58:54.061545', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:54.061545', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879227890580696674, 'uuid_mslong': 10756286823668662967}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '954601e0-7afc-42b7-96fa-c8203895ae62'}, 'oper': 'CREATE', 'request-id': 'req-c09260e8-71da-48cc-a61b-d3fea836d129', 'type': 'analytics_node', 'uuid': '954601e0-7afc-42b7-96fa-c8203895ae62'} 09/01/2025 01:58:54.082 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:54.083 7f2a3d726180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.083 7f2a3d726180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10756286823668662967, "uuid_lslong": 10879227890580696674}, "enable": true, "created": "2025-09-01T01:58:54.061545", "last_modified": "2025-09-01T01:58:54.061545", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "954601e0-7afc-42b7-96fa-c8203895ae62" } deleted = False >> 09/01/2025 01:58:54.083 7f2a3d726180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:55.012 7f2a3d726220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:55.014 7f2a3d726220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:55.014 7f2a3d726220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:55.014 7f2a3d726220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:55.014 7f2a3d726220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:55.014 7f2a3d726220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:55.014 7f2a3d726220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:55.015 7f2a3d726220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:55.015 7f2a3d726220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.015 7f2a3d726220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:55.015 7f2a3d726220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.015 7f2a3d726220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.015 7f2a3d726220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:55.016 7f2a3d726220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:56.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:57.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:58.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:59.014 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:59.015 7f2a380c6860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:59.016 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:00.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:01.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:02.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:03.016 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:03.017 7f2a380c6860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:03.017 7f2a380c6860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.018 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:03.019 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:04.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:05.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:06.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:07.019 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:07.021 7f2a380c6860 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:07.022 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.022 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:07.022 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.022 7f2a380c6860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.022 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:07.022 7f2a380c6860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:08.808 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.623 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:09.624 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:09.625 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:09.626 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:09.626 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:09.626 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:09.626 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:09.626 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:09.626 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:09.687 7f2a380c6860 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'] not found in DB: Unknown id: analytics_alarm_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. 09/01/2025 01:59:09.693 7f2a380c6860 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_alarm_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. url = http://10.0.0.249:8082/analytics-alarm-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_analytics_alarm_node.py remote_ip = 10.0.0.38 body = {"analytics-alarm-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."], "parent_type": "global-system-config", "analytics_alarm_node_ip_address": "10.0.0.38", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:09.718 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:09.711039', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:09.711039', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410244128994570297, 'uuid_mslong': 7346777207241002013}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '65f4fef1-f189-4c1d-8297-e72d7e6d3039'}, 'oper': 'CREATE', 'request-id': 'req-65902cdc-006d-4333-bc45-edc4870831f6', 'type': 'analytics_alarm_node', 'uuid': '65f4fef1-f189-4c1d-8297-e72d7e6d3039'} 09/01/2025 01:59:09.719 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:09.719 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.719 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_alarm_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7346777207241002013, "uuid_lslong": 9410244128994570297}, "enable": true, "created": "2025-09-01T01:59:09.711039", "last_modified": "2025-09-01T01:59:09.711039", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "65f4fef1-f189-4c1d-8297-e72d7e6d3039" } deleted = False >> 09/01/2025 01:59:09.719 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.809 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.981 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:09.982 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:09.983 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:09.984 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:10.035 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:10.035 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:10.036 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:10.037 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:10.038 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:10.038 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:10.039 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:10.047 7f2a380c69a0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'] not found in DB: Unknown id: analytics_alarm_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. 09/01/2025 01:59:10.055 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_alarm_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. url = http://10.0.0.249:8082/analytics-alarm-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_analytics_alarm_node.py remote_ip = 10.0.0.131 body = {"analytics-alarm-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."], "parent_type": "global-system-config", "analytics_alarm_node_ip_address": "10.0.0.131", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:10.101 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:10.086800', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:10.086800', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10503918820682058891, 'uuid_mslong': 906375348849557938}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0c941793-dd59-41b2-91c5-6a83e84b4c8b'}, 'oper': 'CREATE', 'request-id': 'req-5937b566-f775-43d5-a1c9-186dda6980e6', 'type': 'analytics_alarm_node', 'uuid': '0c941793-dd59-41b2-91c5-6a83e84b4c8b'} 09/01/2025 01:59:10.102 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:10.102 7f2a38d072c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.102 7f2a38d072c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_alarm_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 906375348849557938, "uuid_lslong": 10503918820682058891}, "enable": true, "created": "2025-09-01T01:59:10.086800", "last_modified": "2025-09-01T01:59:10.086800", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "0c941793-dd59-41b2-91c5-6a83e84b4c8b" } deleted = False >> 09/01/2025 01:59:10.102 7f2a38d072c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.140 7f2a380c6400 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'] not found in DB: Unknown id: analytics_alarm_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. 09/01/2025 01:59:10.146 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_alarm_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. url = http://10.0.0.249:8082/analytics-alarm-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_analytics_alarm_node.py remote_ip = 10.0.0.249 body = {"analytics-alarm-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."], "parent_type": "global-system-config", "analytics_alarm_node_ip_address": "10.0.0.249", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:10.193 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:10.172514', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:10.172514', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879587110886434360, 'uuid_mslong': 11828816951027650397}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a4286658-5f1f-4f5d-96fc-0ed5b74de638'}, 'oper': 'CREATE', 'request-id': 'req-5363ae59-4436-4596-ac85-f2751a51eb9c', 'type': 'analytics_alarm_node', 'uuid': 'a4286658-5f1f-4f5d-96fc-0ed5b74de638'} 09/01/2025 01:59:10.193 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:10.194 7f2a380c62c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.194 7f2a380c62c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_alarm_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11828816951027650397, "uuid_lslong": 10879587110886434360}, "enable": true, "created": "2025-09-01T01:59:10.172514", "last_modified": "2025-09-01T01:59:10.172514", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a4286658-5f1f-4f5d-96fc-0ed5b74de638" } deleted = False >> 09/01/2025 01:59:10.194 7f2a380c62c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.811 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:11.021 7f2a38155180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:11.022 7f2a38155180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:11.022 7f2a38155180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:11.022 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:11.022 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:11.023 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:12.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:14.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:15.023 7f2a38155180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:15.024 7f2a38155180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:15.024 7f2a38155180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:15.024 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:15.024 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:15.024 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:15.024 7f2a38155180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:15.025 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:16.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:17.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:18.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:19.025 7f2a38155180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:19.026 7f2a38155180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:19.026 7f2a38155180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:19.027 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:19.027 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:19.027 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:19.027 7f2a38155180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:19.028 7f2a38155180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:20.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:21.782 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:21.783 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:21.784 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:21.784 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:21.784 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:21.784 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:21.784 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:21.785 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:21.785 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:21.785 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:21.785 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:21.786 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:21.786 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:21.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.844 7f2a38155180 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'] not found in DB: Unknown id: analytics_snmp_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. 09/01/2025 01:59:21.849 7f2a38155180 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_snmp_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-3. url = http://10.0.0.249:8082/analytics-snmp-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_analytics_snmp_node.py remote_ip = 10.0.0.131 body = {"analytics-snmp-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."], "parent_type": "global-system-config", "analytics_snmp_node_ip_address": "10.0.0.131", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-3."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:21.875 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:21.867078', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:21.867078', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13422430401004267031, 'uuid_mslong': 671617895336398686}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'}, 'oper': 'CREATE', 'request-id': 'req-0bcb5258-1518-4af3-bc32-66c2afda7f6a', 'type': 'analytics_snmp_node', 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'} 09/01/2025 01:59:21.875 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:21.876 7f2a33db0e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.876 7f2a33db0e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_snmp_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 671617895336398686, "uuid_lslong": 13422430401004267031}, "enable": true, "created": "2025-09-01T01:59:21.867078", "last_modified": "2025-09-01T01:59:21.867078", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "095210ea-dc52-4f5e-ba46-0d55aa436617" } deleted = False >> 09/01/2025 01:59:21.876 7f2a33db0e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.141 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:22.141 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:22.142 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:22.143 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:22.144 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:22.145 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:22.146 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 01:59:22.175 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 01:59:22.176 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 01:59:22.177 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 01:59:22.202 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'] not found in DB: Unknown id: analytics_snmp_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. 09/01/2025 01:59:22.207 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_snmp_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-2. url = http://10.0.0.249:8082/analytics-snmp-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_analytics_snmp_node.py remote_ip = 10.0.0.38 body = {"analytics-snmp-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."], "parent_type": "global-system-config", "analytics_snmp_node_ip_address": "10.0.0.38", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-2."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:22.234 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:22.227040', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.227040', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10043001507123339914, 'uuid_mslong': 13054484782216790713}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'}, 'oper': 'CREATE', 'request-id': 'req-cc52ae74-0d4d-419c-a865-1eba05a95d49', 'type': 'analytics_snmp_node', 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'} 09/01/2025 01:59:22.235 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:22.236 7f2a33db0a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.236 7f2a33db0a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_snmp_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13054484782216790713, "uuid_lslong": 10043001507123339914}, "enable": true, "created": "2025-09-01T01:59:22.227040", "last_modified": "2025-09-01T01:59:22.227040", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "b52ad8be-69c6-42b9-8b5f-e8a91ecca28a" } deleted = False >> 09/01/2025 01:59:22.236 7f2a33db0a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.273 7f2a380c65e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'] not found in DB: Unknown id: analytics_snmp_node default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. 09/01/2025 01:59:22.278 7f2a380c65e0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_snmp_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-ansible-os-3995-1. url = http://10.0.0.249:8082/analytics-snmp-nodes operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_analytics_snmp_node.py remote_ip = 10.0.0.249 body = {"analytics-snmp-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."], "parent_type": "global-system-config", "analytics_snmp_node_ip_address": "10.0.0.249", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-ansible-os-3995-1."}} domain = default-domain project = admin user = admin >> 09/01/2025 01:59:22.307 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:22.300301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.300301', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10295412184530401449, 'uuid_mslong': 11790706739729089604}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'}, 'oper': 'CREATE', 'request-id': 'req-79a7348d-975b-4e66-ac0e-d0c0e9ecf4b1', 'type': 'analytics_snmp_node', 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'} 09/01/2025 01:59:22.312 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:22.313 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.314 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_snmp_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11790706739729089604, "uuid_lslong": 10295412184530401449}, "enable": true, "created": "2025-09-01T01:59:22.300301", "last_modified": "2025-09-01T01:59:22.300301", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a3a1014f-585b-4444-8ee0-a6d59b29b8a9" } deleted = False >> 09/01/2025 01:59:22.314 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:23.026 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:23.028 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:23.029 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.029 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:23.029 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.029 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.029 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:23.029 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:24.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:25.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:26.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:27.028 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:27.029 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:27.030 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.031 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:27.031 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.031 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.031 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:27.031 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:28.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:29.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:30.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:31.031 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:31.032 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:31.032 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:31.032 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:31.032 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:31.032 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:31.032 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:31.033 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:32.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:33.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:34.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:35.034 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:35.035 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:35.035 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:35.035 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:35.036 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:36.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:38.806 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:39.036 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:39.037 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.038 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.039 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:39.039 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.807 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:40.809 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.811 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:42.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:43.038 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:43.040 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:43.040 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:43.040 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:43.040 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:43.040 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:43.040 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:43.041 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:44.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:45.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:46.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:47.040 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:47.042 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.043 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.043 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:47.043 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:48.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:49.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:50.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:51.043 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:51.044 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:51.044 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:51.044 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:51.044 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:51.044 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:51.044 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:51.045 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:52.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:53.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:54.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:55.044 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:55.045 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:55.045 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.046 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.047 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:55.047 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:56.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:57.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:58.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:59.046 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:59.047 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.048 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.049 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:59.049 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:00.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:01.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:02.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:03.049 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:03.050 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:03.051 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.051 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:03.051 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.051 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.051 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:03.051 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:04.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:06.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:07.051 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:07.052 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:07.052 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:07.052 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:07.052 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:07.052 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:07.053 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:07.053 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:07.053 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.053 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:07.053 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.054 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.054 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:07.054 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:08.807 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.809 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:10.810 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:11.053 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:11.054 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:11.054 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:11.054 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:11.054 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:11.054 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:11.054 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:11.055 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:12.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:14.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:15.055 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:15.056 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:15.056 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:15.056 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:15.056 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:15.057 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:16.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:17.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:18.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:19.057 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:19.058 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:19.058 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:19.058 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:19.058 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:19.058 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:19.058 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:19.059 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:20.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:21.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:22.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:23.059 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:23.060 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:23.060 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:23.060 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:23.061 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:24.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:25.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:26.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:27.061 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:27.062 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:27.062 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:27.062 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:27.062 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:27.062 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:27.062 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:27.063 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:28.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:29.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:30.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:31.063 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:31.064 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:31.064 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:31.064 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:31.064 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:31.064 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:31.064 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:31.065 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:32.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:34.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:35.065 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:35.066 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:35.066 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:35.066 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:35.066 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:35.067 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:36.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:38.808 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:39.067 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:39.068 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:39.068 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:39.068 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:39.068 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:39.068 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:39.068 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:39.069 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.810 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:40.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:42.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:43.069 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:43.070 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:43.071 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.072 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.072 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:43.072 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:44.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:45.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:46.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:47.071 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:47.072 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:47.072 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:47.072 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:47.072 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:47.072 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:47.072 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:47.073 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:48.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:49.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:50.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:51.073 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:51.074 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:51.074 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:51.074 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.075 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.076 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:51.076 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:52.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:53.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:54.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:55.075 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:55.076 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:55.076 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:55.076 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:55.076 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:55.076 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:55.076 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:55.077 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:56.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:57.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:58.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:59.077 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:59.078 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:59.078 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:59.078 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:59.078 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:59.078 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:59.078 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:59.079 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:00.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:01.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:02.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:03.079 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:03.080 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:03.080 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:03.080 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:03.080 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:03.080 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:03.080 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:03.081 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:04.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:05.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:06.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:07.080 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.082 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:07.083 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.083 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.083 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:07.083 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:08.809 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:09.810 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:01:10.355 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:01:10.356 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:01:10.357 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:01:10.358 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:01:10.414 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 02:01:10.414 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 02:01:10.414 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 02:01:10.414 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 02:01:10.414 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 02:01:10.414 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 02:01:10.415 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 02:01:10.415 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 02:01:10.463 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dee9192c-e41b-4855-8333-2236491cfcf8 object_type = global_system_config identifier_name = default-global-system-config url = http://10.0.0.249:8082/global-system-config/dee9192c-e41b-4855-8333-2236491cfcf8 operation = http_put useragent = an-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.57 body = {"global-system-config": {"uuid": "dee9192c-e41b-4855-8333-2236491cfcf8", "ip_fabric_subnets": {"subnet": [{"ip_prefix": "10.20.0.0", "ip_prefix_len": 24}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 02:01:10.477 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-d6447694-7a63-4e16-8ab1-f2203b502079', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 02:01:10.571 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'] not found in DB: Unknown id: virtual_router default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2 09/01/2025 02:01:10.576 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_router identifier_name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2 url = http://10.0.0.249:8082/virtual-routers operation = post useragent = an-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.57 body = {"virtual-router": {"uuid": null, "fq_name": ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2"], "parent_type": "global-system-config", "virtual_router_dpdk_enabled": false, "virtual_router_ip_address": "10.20.0.192", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "an-jenkins-deploy-platform-ansible-os-3995-2"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:01:10.601 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T02:01:10.595372', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.595372', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11952340681741785153, 'uuid_mslong': 6586627194885784908}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.192'}, 'oper': 'CREATE', 'request-id': 'req-681edb8a-0de7-491e-8459-bf06930acfe8', 'type': 'virtual_router', 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841'} 09/01/2025 02:01:10.602 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:01:10.602 7f2a38d07900 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.602 7f2a38d07900 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = an-jenkins-deploy-platform-ansible-os-3995-2 elements = { display_name : "an-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6586627194885784908, "uuid_lslong": 11952340681741785153}, "enable": true, "created": "2025-09-01T02:01:10.595372", "last_modified": "2025-09-01T02:01:10.595372", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.192" } deleted = False >> 09/01/2025 02:01:10.602 7f2a38d07900 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.698 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:01:10.698 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:01:10.698 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:01:10.698 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:01:10.699 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:01:10.699 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:01:10.699 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:01:10.699 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:01:10.811 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:10.842 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field access_control_lists in resource virtual-network: 'access_control_lists' 09/01/2025 02:01:10.842 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field floating_ip_pools in resource virtual-network: 'floating_ip_pools' 09/01/2025 02:01:10.842 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alias_ip_pools in resource virtual-network: 'alias_ip_pools' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instances in resource virtual-network: 'routing_instances' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bridge_domains in resource virtual-network: 'bridge_domains' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_object_refs in resource virtual-network: 'security_logging_object_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_refs in resource virtual-network: 'qos_config_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-network: 'network_ipam_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_policy_refs in resource virtual-network: 'network_policy_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_refs in resource virtual-network: 'virtual_network_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_health_check_refs in resource virtual-network: 'service_health_check_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_table_refs in resource virtual-network: 'route_table_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field multicast_policy_refs in resource virtual-network: 'multicast_policy_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgpvpn_refs in resource virtual-network: 'bgpvpn_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_map_refs in resource virtual-network: 'intent_map_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_refs in resource virtual-network: 'routing_policy_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-network: 'tag_refs' 09/01/2025 02:01:10.843 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource virtual-network: 'virtual_network_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource virtual-network: 'virtual_machine_interface_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-network: 'instance_ip_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-network: 'physical_router_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_tuple_back_refs in resource virtual-network: 'port_tuple_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_router_back_refs in resource virtual-network: 'logical_router_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_node_back_refs in resource virtual-network: 'flow_node_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field firewall_rule_back_refs in resource virtual-network: 'firewall_rule_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnect_back_refs in resource virtual-network: 'data_center_interconnect_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabric_back_refs in resource virtual-network: 'fabric_back_refs' 09/01/2025 02:01:10.844 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field host_based_service_back_refs in resource virtual-network: 'host_based_service_back_refs' 09/01/2025 02:01:10.895 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'] not found in DB: Unknown id: virtual_machine_interface default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 09/01/2025 02:01:10.901 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_machine_interface identifier_name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 url = http://10.0.0.249:8082/virtual-machine-interfaces operation = post useragent = an-jenkins-deploy-platform-ansible-os-3995-2:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.57 body = {"virtual-machine-interface": {"uuid": null, "fq_name": ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"], "parent_type": "virtual-router", "virtual_machine_interface_disable_policy": true, "virtual_network_refs": [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "vhost0"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:01:10.940 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:01:10.925113', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.925113', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10735908003063543833, 'uuid_mslong': 17648289949859202616}}, 'parent_type': 'virtual-router', 'parent_uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:f4:eb:52:34:6a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.940 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:01:10.940 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.941 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17648289949859202616, "uuid_lslong": 10735908003063543833}, "enable": true, "created": "2025-09-01T02:01:10.925113", "last_modified": "2025-09-01T02:01:10.925113", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "f4eb5234-6a74-4638-94fd-9b728fe03819"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:f4:eb:52:34:6a"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:01:10.941 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.950 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = f4eb5234-6a74-4638-94fd-9b728fe03819 object_type = virtual_machine_interface identifier_name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 url = http://127.0.0.1/ref-update operation = ref-update domain = default-domain >> 09/01/2025 02:01:10.967 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.971 7f2a39836180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.971 7f2a39836180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17648289949859202616, "uuid_lslong": 10735908003063543833}, "enable": true, "created": "2025-09-01T02:01:10.925113", "last_modified": "2025-09-01T02:01:10.956445", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7"}]uuid : "f4eb5234-6a74-4638-94fd-9b728fe03819"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:f4:eb:52:34:6a"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:01:10.971 7f2a39836180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.082 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:11.083 7f2a39836d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:11.083 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:11.083 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:11.083 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:11.083 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:11.084 7f2a39836d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.482 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-8b202558-6c4f-4e38-a46f-f0bfe3ec33be', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 02:01:11.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:12.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:14.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:15.084 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:15.086 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:15.086 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:15.086 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:15.086 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:15.086 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:15.086 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:15.087 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:16.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:17.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:18.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:19.087 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:19.088 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:19.088 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:19.088 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:19.088 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:19.089 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:20.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:21.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:22.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:23.088 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:23.090 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:23.091 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.091 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:23.091 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.091 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.091 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:23.091 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:24.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:25.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:26.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:27.091 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:27.093 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.094 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:27.094 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.094 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.094 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:27.094 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:28.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:29.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:30.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:31.093 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:31.095 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:31.096 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.096 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:31.096 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.096 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.096 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:31.096 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:32.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:33.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:34.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:35.095 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:35.103 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:35.103 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:35.103 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:35.104 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:36.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:38.811 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:39.104 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:39.105 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.106 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:39.106 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.106 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.106 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:39.106 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:40.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:42.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:43.106 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:43.107 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:43.107 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:43.107 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:43.107 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:43.107 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:43.107 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:43.108 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:44.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:45.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:46.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:47.108 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:47.109 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:47.109 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:47.110 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:47.110 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:47.110 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:47.110 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:47.110 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:47.110 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.111 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:47.111 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.111 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.111 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:47.111 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:48.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:49.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:50.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:51.111 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:51.111 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:51.112 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.113 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:51.113 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.113 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.113 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:51.113 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:52.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:53.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:54.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:55.113 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.114 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:55.115 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.115 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.115 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:55.115 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:56.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:57.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:58.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:59.115 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:59.116 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:59.116 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:59.116 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:59.117 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:00.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:01.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.805 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 09/01/2025 02:02:02.806 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 09/01/2025 02:02:02.807 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 09/01/2025 02:02:02.808 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 09/01/2025 02:02:02.808 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 09/01/2025 02:02:02.808 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 09/01/2025 02:02:02.808 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 09/01/2025 02:02:02.808 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 09/01/2025 02:02:02.808 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 09/01/2025 02:02:02.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 09/01/2025 02:02:02.863 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 09/01/2025 02:02:02.864 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 09/01/2025 02:02:02.959 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'] not found in DB: Unknown id: virtual_router default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1 09/01/2025 02:02:02.965 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_router identifier_name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1 url = http://10.0.0.249:8082/virtual-routers operation = post useragent = an-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.41 body = {"virtual-router": {"uuid": null, "fq_name": ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1"], "parent_type": "global-system-config", "virtual_router_dpdk_enabled": false, "virtual_router_ip_address": "10.20.0.18", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "an-jenkins-deploy-platform-ansible-os-3995-1"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:02:02.992 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T02:02:02.985199', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:02.985199', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10627036196304171688, 'uuid_mslong': 16146534956283939804}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.18'}, 'oper': 'CREATE', 'request-id': 'req-7f54cc47-40c2-4259-85af-4adeb8e3ee0f', 'type': 'virtual_router', 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8'} 09/01/2025 02:02:02.992 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:02:02.992 7f2a380c6c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.993 7f2a380c6c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = an-jenkins-deploy-platform-ansible-os-3995-1 elements = { display_name : "an-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16146534956283939804, "uuid_lslong": 10627036196304171688}, "enable": true, "created": "2025-09-01T02:02:02.985199", "last_modified": "2025-09-01T02:02:02.985199", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.18" } deleted = False >> 09/01/2025 02:02:02.993 7f2a380c6c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.090 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:02:03.090 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:02:03.090 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:02:03.090 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:02:03.090 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:02:03.090 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:02:03.091 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:02:03.091 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:02:03.117 7f2a381550e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:03.118 7f2a381550e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:03.119 7f2a381550e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:03.119 7f2a381550e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:03.119 7f2a381550e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:03.119 7f2a381550e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:03.119 7f2a381550e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:03.119 7f2a381550e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:03.120 7f2a381550e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.120 7f2a381550e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:03.120 7f2a381550e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.120 7f2a381550e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.120 7f2a381550e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:03.120 7f2a381550e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field access_control_lists in resource virtual-network: 'access_control_lists' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field floating_ip_pools in resource virtual-network: 'floating_ip_pools' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alias_ip_pools in resource virtual-network: 'alias_ip_pools' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instances in resource virtual-network: 'routing_instances' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bridge_domains in resource virtual-network: 'bridge_domains' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_object_refs in resource virtual-network: 'security_logging_object_refs' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_refs in resource virtual-network: 'qos_config_refs' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-network: 'network_ipam_refs' 09/01/2025 02:02:03.235 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_policy_refs in resource virtual-network: 'network_policy_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_refs in resource virtual-network: 'virtual_network_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_health_check_refs in resource virtual-network: 'service_health_check_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_table_refs in resource virtual-network: 'route_table_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field multicast_policy_refs in resource virtual-network: 'multicast_policy_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgpvpn_refs in resource virtual-network: 'bgpvpn_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_map_refs in resource virtual-network: 'intent_map_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_refs in resource virtual-network: 'routing_policy_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-network: 'tag_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource virtual-network: 'virtual_network_back_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource virtual-network: 'virtual_machine_interface_back_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-network: 'instance_ip_back_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-network: 'physical_router_back_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_tuple_back_refs in resource virtual-network: 'port_tuple_back_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_router_back_refs in resource virtual-network: 'logical_router_back_refs' 09/01/2025 02:02:03.236 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_node_back_refs in resource virtual-network: 'flow_node_back_refs' 09/01/2025 02:02:03.237 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field firewall_rule_back_refs in resource virtual-network: 'firewall_rule_back_refs' 09/01/2025 02:02:03.237 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnect_back_refs in resource virtual-network: 'data_center_interconnect_back_refs' 09/01/2025 02:02:03.237 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabric_back_refs in resource virtual-network: 'fabric_back_refs' 09/01/2025 02:02:03.237 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field host_based_service_back_refs in resource virtual-network: 'host_based_service_back_refs' 09/01/2025 02:02:03.288 7f2a380c6040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'] not found in DB: Unknown id: virtual_machine_interface default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 09/01/2025 02:02:03.294 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_machine_interface identifier_name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 url = http://10.0.0.249:8082/virtual-machine-interfaces operation = post useragent = an-jenkins-deploy-platform-ansible-os-3995-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.41 body = {"virtual-machine-interface": {"uuid": null, "fq_name": ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"], "parent_type": "virtual-router", "virtual_machine_interface_disable_policy": true, "virtual_network_refs": [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "vhost0"}} domain = default-domain project = admin user = admin >> 09/01/2025 02:02:03.331 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:03.316910', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:03.316910', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12779270811540185455, 'uuid_mslong': 492471125198063140}}, 'parent_type': 'virtual-router', 'parent_uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:06:d5:9b:e1:0f']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.332 7f2a38cf0720 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:02:03.332 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.332 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 492471125198063140, "uuid_lslong": 12779270811540185455}, "enable": true, "created": "2025-09-01T02:02:03.316910", "last_modified": "2025-09-01T02:02:03.316910", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "06d59be1-0f91-4624-b159-17187a5ee16f"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:06:d5:9b:e1:0f"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:02:03.332 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.340 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 06d59be1-0f91-4624-b159-17187a5ee16f object_type = virtual_machine_interface identifier_name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 url = http://127.0.0.1/ref-update operation = ref-update domain = default-domain >> 09/01/2025 02:02:03.360 7f2a38cf0720 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.363 7f2a38acfb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.364 7f2a38acfb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 492471125198063140, "uuid_lslong": 12779270811540185455}, "enable": true, "created": "2025-09-01T02:02:03.316910", "last_modified": "2025-09-01T02:02:03.346223", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7"}]uuid : "06d59be1-0f91-4624-b159-17187a5ee16f"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:06:d5:9b:e1:0f"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:02:03.364 7f2a38acfb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:04.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:05.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:06.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:07.119 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:07.120 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:07.120 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:07.120 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:07.120 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:07.121 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:08.811 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:10.814 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:11.120 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:11.122 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:11.122 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:11.122 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:11.122 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:11.122 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:11.122 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:11.123 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:02:12.557 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:02:12.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:13.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:14.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:15.122 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.124 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.125 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:15.125 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:16.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:17.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:18.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:19.125 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:19.126 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:19.127 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.127 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:19.127 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.127 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.127 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:19.127 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:20.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:21.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:22.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:23.126 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:23.127 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:23.127 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:23.127 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:23.128 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:24.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:25.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:26.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:27.128 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:27.129 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:27.129 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:27.129 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:27.129 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:27.129 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:27.129 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:27.130 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:28.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:29.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:30.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:31.129 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:31.131 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:31.132 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:31.132 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:31.132 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:31.132 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:31.132 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:31.132 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:31.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:32.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:33.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:34.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:35.131 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:35.135 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:35.135 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:35.135 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:35.136 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:35.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:36.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:38.812 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:39.136 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:39.137 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:39.138 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:39.138 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:39.138 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:39.138 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:39.138 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:39.138 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:39.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:40.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:41.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:42.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:43.137 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:43.139 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:43.140 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:43.140 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:43.140 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:43.140 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:43.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:44.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:45.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:46.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:47.139 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:47.140 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:47.141 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:47.142 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:47.142 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:47.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:48.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:49.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:50.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:51.141 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:51.142 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:51.143 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:51.144 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:51.144 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:51.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:52.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:53.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:54.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:55.144 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:55.145 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:55.146 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:55.146 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:55.146 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:55.146 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:55.146 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:55.146 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:55.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:56.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:57.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:58.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:59.146 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:59.147 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:59.148 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:59.149 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:59.149 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:59.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:00.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:01.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:02.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:03.148 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:03.149 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:03.150 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:03.151 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:03.151 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:03.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:04.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:06.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:07.150 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:07.151 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:07.152 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:07.152 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:07.152 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:07.152 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:07.152 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:07.152 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:07.153 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:07.153 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:07.153 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:07.153 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:07.153 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:07.153 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:10.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:11.153 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:11.155 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:11.156 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:11.156 7f2a380c64a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:11.156 7f2a380c64a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:11.156 7f2a380c64a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:11.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:12.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.435 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:03:13.435 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:03:13.435 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:03:13.435 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:03:13.436 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:03:13.436 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:03:13.436 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:03:13.436 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:03:13.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:14.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:15.155 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:15.157 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:15.157 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:15.157 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:15.157 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:15.157 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:15.157 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:15.158 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:15.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:16.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:17.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:18.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:19.158 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:19.159 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:19.160 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:19.161 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:19.161 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:19.161 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:19.161 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:19.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:20.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:21.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:22.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:23.160 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:23.161 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:23.162 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:23.163 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:23.163 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:23.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:24.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:25.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:26.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:27.162 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:27.163 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:27.164 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:27.165 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:27.165 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:27.165 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:27.165 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:27.165 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:27.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:28.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:29.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:30.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:31.165 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:31.166 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:31.166 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:31.166 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:31.166 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:31.166 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:31.166 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:31.167 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:31.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:32.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:33.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:34.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:35.166 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:35.167 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.168 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.169 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:35.169 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:36.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:38.813 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:39.168 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:39.169 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:39.169 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:39.169 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:39.170 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:39.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:40.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:42.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:43.170 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:43.172 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:43.172 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:43.172 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:43.172 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:43.172 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:43.172 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:43.173 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:43.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:44.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:45.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:46.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:47.173 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:47.174 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:47.174 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:47.174 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:47.174 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:47.175 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:47.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:48.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:49.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:50.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:51.175 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:51.176 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:51.176 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:51.177 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:51.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:52.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:53.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:54.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:55.177 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:55.178 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:55.178 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:55.179 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:55.180 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:55.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:56.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:57.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:58.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:59.179 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:59.180 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:59.180 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:59.180 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:59.181 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:59.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:00.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:01.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:02.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:03.181 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:03.182 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:03.182 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:03.182 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:03.182 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:03.182 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:03.182 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:03.183 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:03.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:04.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:06.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:07.183 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:07.184 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:07.184 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:07.184 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:07.184 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:07.184 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:07.184 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:07.185 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.815 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:10.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:11.185 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:11.187 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:11.187 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:11.187 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:11.188 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:11.188 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:11.188 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:11.188 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:11.188 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:11.188 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:11.189 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:11.189 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:11.189 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:11.189 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:11.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:12.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:13.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:14.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:15.188 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:15.189 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:15.189 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:15.189 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:15.189 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:15.189 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:15.189 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:15.190 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:15.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:16.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:17.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:18.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:19.190 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:19.191 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:19.191 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:19.191 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:19.192 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:19.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:20.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:21.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:22.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:23.192 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:23.194 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:23.194 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:23.194 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:23.194 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:23.194 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:23.194 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:23.195 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:23.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:24.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:25.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:26.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:27.194 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:27.195 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:27.195 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:27.195 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:27.195 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:27.195 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:27.196 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:27.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:28.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:29.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:30.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:31.196 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:31.197 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.198 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:31.199 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.199 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.199 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:31.199 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:31.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:32.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:34.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:35.199 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:35.200 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:35.200 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:35.200 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:35.201 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:36.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:38.816 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:39.201 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:39.202 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.203 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:39.204 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.204 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.204 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:39.204 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:40.820 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:42.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:43.204 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:43.205 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:43.205 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.206 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:43.207 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.207 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.207 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:43.207 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:43.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:44.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:45.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:46.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:47.207 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:47.208 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:47.208 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:47.209 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:47.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:48.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:49.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:50.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:51.209 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:51.210 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:51.210 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:51.210 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:51.211 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:52.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:53.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:54.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:55.211 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:55.212 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:55.212 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:55.212 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:55.212 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:55.212 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:55.212 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:55.213 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:56.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:57.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:58.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:59.213 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.215 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:59.216 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.216 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.216 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:59.216 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:00.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:01.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:02.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:03.215 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:03.216 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:03.216 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:03.216 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:03.216 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:03.216 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:03.217 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:04.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:05.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:06.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:07.216 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.218 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:07.219 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.219 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.219 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:07.219 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.817 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:10.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:11.219 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:11.220 7f2a3a214cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:11.220 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:11.220 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:11.221 7f2a3a214cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:12.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:14.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:15.046 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:05:15.047 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:05:15.221 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:15.222 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:15.222 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:15.222 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:15.222 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:15.223 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:16.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:17.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:18.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:19.223 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:19.224 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:19.224 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:19.224 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:19.225 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:20.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:21.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:22.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:23.225 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:23.226 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:23.226 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:23.227 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:24.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:26.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:27.227 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:27.228 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:27.229 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.229 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:27.229 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.229 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.229 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:27.229 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:28.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:29.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:30.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:31.228 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.230 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.231 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:31.231 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:32.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:34.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:35.230 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:35.231 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:35.231 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:35.231 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:35.232 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:36.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:38.818 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:39.232 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:39.233 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:39.233 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:39.234 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:40.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:42.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:43.234 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:43.235 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:43.235 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:43.235 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:43.235 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:43.235 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:43.235 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:43.236 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:44.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:45.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:46.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:47.235 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:47.236 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:47.237 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.238 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:47.238 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.238 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.238 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:47.238 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:48.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:49.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:50.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:51.238 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:51.239 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:51.240 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.240 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:51.240 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.240 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.240 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:51.240 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:52.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:53.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:54.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:55.239 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:55.240 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:55.240 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:55.240 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:55.241 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:56.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:57.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:58.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:59.241 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:59.243 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.244 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:59.244 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.244 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.244 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:59.244 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:00.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:01.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:02.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:03.244 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:03.245 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:03.245 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:03.245 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:03.245 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:03.245 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:03.245 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:03.246 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:04.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:06.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:07.245 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:07.247 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:07.247 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:07.247 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:07.247 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:07.247 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:07.247 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:07.248 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:09.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:10.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:11.248 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:11.249 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:11.249 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.250 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.251 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:11.251 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:12.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:14.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:15.250 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:15.251 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:15.251 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:15.251 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:15.251 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:15.251 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:15.251 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:15.252 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:15.930 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:06:15.931 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:06:16.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:18.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:19.251 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:19.253 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.254 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:19.254 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.254 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.254 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:19.254 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:20.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:22.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:23.254 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:23.255 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:23.255 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:23.255 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:23.255 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:23.256 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:24.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:26.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:27.257 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:27.258 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:27.258 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:27.258 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:27.258 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:27.258 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:27.258 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:27.259 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:28.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:29.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:30.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:31.259 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:31.260 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:31.260 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:31.261 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:32.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:34.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:35.261 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:35.263 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.264 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:35.264 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.264 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.264 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:35.264 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:36.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:38.819 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:39.263 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:39.265 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:39.265 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:39.265 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:39.265 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:39.265 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:39.265 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:39.266 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:40.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:42.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:43.265 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:43.266 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:43.266 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.267 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:43.268 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:44.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:45.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:46.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:47.268 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:47.270 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:47.270 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:47.270 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:47.270 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:47.270 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:47.270 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:47.271 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:48.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:49.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:50.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:51.270 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:51.271 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:51.271 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:51.271 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:51.271 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:51.272 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:52.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:53.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:54.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:55.272 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:55.273 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:55.274 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:55.274 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:55.274 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:55.274 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:55.274 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:55.274 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:55.275 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.275 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:55.275 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.275 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.275 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:55.275 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:56.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:57.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:58.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:59.274 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:59.275 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:59.275 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:59.275 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:59.275 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:59.275 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:59.275 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:59.276 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:00.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:01.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:02.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:03.275 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:03.277 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:03.277 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:03.277 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:03.277 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:03.277 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:03.277 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:03.278 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:04.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:05.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:06.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:07.278 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:07.279 7f2a380c6040 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:07.279 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:07.279 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:07.279 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:07.279 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:07.279 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:07.280 7f2a380c6040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.821 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:09.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:10.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:11.280 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.282 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:11.283 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.283 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.283 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:11.283 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:12.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:14.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:15.282 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:15.283 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:15.284 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.285 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.285 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:15.285 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:17.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:18.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:19.284 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.286 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:19.287 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.287 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.287 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:19.287 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:20.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:21.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:22.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:23.287 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:23.288 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.289 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:23.290 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:24.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:25.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:26.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:27.289 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:27.290 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:27.290 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:27.290 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:27.290 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:27.290 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:27.291 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:28.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:29.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:30.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:31.291 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:31.292 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:31.292 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:31.292 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:31.292 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:31.292 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:31.292 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:31.293 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:32.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:33.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:34.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:35.293 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:35.294 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:35.294 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:35.294 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:35.295 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:36.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:38.822 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:39.295 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:39.296 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:39.296 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:39.296 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:39.296 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:39.296 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:39.296 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:39.297 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:40.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:41.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:42.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:43.297 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:43.298 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:43.298 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:43.298 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:43.298 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:43.298 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:43.298 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:43.299 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:43.299 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.299 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:43.299 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.299 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.299 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:43.300 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:44.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:45.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:46.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:47.299 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:47.300 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.301 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:47.302 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.302 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.302 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:47.302 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:48.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:49.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:50.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:51.301 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:51.302 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:51.302 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:51.302 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.303 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.304 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:51.304 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:52.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:53.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:54.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:55.303 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:55.304 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:55.304 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:55.304 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:55.304 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:55.304 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:55.304 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:55.305 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:55.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:56.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:57.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:58.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:59.305 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:59.306 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.307 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.308 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:59.308 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:00.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:01.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:02.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:03.308 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:03.309 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:03.309 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:03.309 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:03.310 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:05.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:06.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:07.310 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:07.311 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:07.312 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.312 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:07.312 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.312 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.312 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:07.312 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.823 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:09.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:10.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:11.312 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:11.313 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:11.313 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.314 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:11.315 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:12.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:13.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:14.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:15.315 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:15.316 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:15.316 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:15.316 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:15.316 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.317 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.318 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:15.318 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:16.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.535 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:08:17.536 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:08:17.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:18.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:19.316 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:19.318 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:19.318 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:19.318 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:19.318 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:19.318 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:19.318 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:19.319 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:20.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:21.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:22.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:23.319 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:23.320 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:23.320 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:23.320 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:23.320 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:23.320 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:23.321 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:24.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:25.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:26.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:27.322 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.323 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:27.324 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.324 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.324 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:27.324 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:28.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:30.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:31.323 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:31.324 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:31.324 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:31.324 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:31.325 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:32.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:33.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:34.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:35.326 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:35.327 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:35.327 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:35.327 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:35.327 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:35.327 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:35.327 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:35.328 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:36.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:38.824 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:39.328 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:39.329 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:39.330 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.330 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:39.330 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.330 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.330 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:39.330 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:40.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:42.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:43.330 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:43.331 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:43.331 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:43.331 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:43.332 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:44.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:45.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:46.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:47.332 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:47.334 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:47.334 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:47.334 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:47.334 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:47.334 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:47.334 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:47.335 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:48.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:49.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:50.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:51.335 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:51.337 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:51.337 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:51.337 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:51.337 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:51.337 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:51.337 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:51.338 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:52.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:54.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:55.337 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:55.338 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.339 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.340 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:55.340 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:56.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:57.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:58.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:59.339 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:59.340 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.341 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:59.342 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.342 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.342 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:59.342 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:00.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:01.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:02.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:03.342 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:03.343 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:03.343 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:03.343 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:03.343 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:03.343 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:03.343 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:03.344 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:04.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:06.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:07.344 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:07.345 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:07.345 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:07.345 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:07.345 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:07.345 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:07.345 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:07.346 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.825 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:10.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:11.346 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:11.347 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:11.347 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:11.347 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:11.347 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:11.348 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:12.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:13.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:14.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:15.349 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:15.349 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:15.350 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.351 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:15.351 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.351 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.351 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:15.351 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:16.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:18.403 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:09:18.403 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:09:18.403 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:09:18.403 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:09:18.404 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:09:18.404 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:09:18.404 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:09:18.404 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:09:18.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:19.350 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:19.351 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:19.351 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:19.352 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:20.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:21.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:22.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:23.352 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:23.354 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:23.354 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:23.354 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:23.354 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:23.354 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:23.354 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:23.355 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:24.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:25.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:26.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:27.355 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:27.356 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:27.356 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:27.356 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.357 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:27.358 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:28.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:30.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:31.357 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:31.358 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:31.358 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:31.358 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.359 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:31.360 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:32.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:33.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:34.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:35.359 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:35.361 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:35.361 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:35.361 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:35.361 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:35.361 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:35.361 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:35.362 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:36.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:38.826 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:39.362 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:39.363 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:39.363 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:39.363 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:39.363 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:39.363 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:39.363 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:39.364 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:40.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:41.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:42.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:43.364 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:43.365 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:43.365 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:43.365 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:43.365 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:43.365 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:43.365 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:43.366 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:44.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:45.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:46.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:47.366 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:47.367 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:47.367 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:47.367 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:47.367 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:47.367 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:47.367 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:47.368 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:48.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:49.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:50.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:51.368 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.369 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:51.370 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.370 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.370 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:51.370 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:52.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:53.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:54.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:55.369 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:55.370 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:55.370 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.371 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.372 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:55.372 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:56.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:57.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:58.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:59.371 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:59.372 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:59.372 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:59.373 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.374 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.374 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:59.374 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:01.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:02.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:03.373 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:03.374 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.375 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.376 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:03.376 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:04.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:05.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:06.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:07.376 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:07.377 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:07.377 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:07.377 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:07.377 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:07.377 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:07.377 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:07.378 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.827 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:09.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:11.378 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:11.380 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:11.380 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:11.380 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:11.380 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:11.380 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:11.380 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:11.381 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:12.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:13.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:14.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:15.381 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:15.382 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:15.382 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:15.382 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:15.382 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:15.382 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:15.383 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:16.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:17.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:18.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:19.383 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:19.384 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:19.385 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:19.385 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:19.385 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:19.385 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:19.385 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:19.385 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:19.386 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.386 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:19.386 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.386 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.386 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:19.386 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:20.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:21.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:22.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:23.386 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:23.387 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:23.387 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:23.387 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:23.388 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:23.388 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:23.388 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:23.388 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:23.388 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.389 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:23.389 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.389 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.389 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:23.389 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:24.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:25.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:26.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:27.388 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:27.389 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:27.389 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:27.389 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:27.389 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:27.389 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:27.389 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:27.390 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:28.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:29.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:31.389 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:31.390 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.391 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:31.392 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:32.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:33.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:35.391 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:35.392 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.393 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:35.394 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.394 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.394 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:35.394 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:36.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.828 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:39.394 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:39.395 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:39.396 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.396 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:39.396 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.396 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.396 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:39.396 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:40.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:41.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:43.396 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:43.398 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.399 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:43.399 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.399 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.399 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:43.399 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:44.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:45.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:46.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:47.399 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:47.400 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:47.400 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:47.400 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:47.401 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:48.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:49.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:50.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:51.402 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:51.403 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:51.403 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:51.403 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.403 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:51.403 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.403 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:51.404 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:52.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:53.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:54.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:55.403 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:55.405 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.406 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:55.406 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.406 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.406 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:55.406 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:56.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:57.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:59.406 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:59.407 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:59.407 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:59.407 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.407 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:59.407 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:59.408 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:00.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:01.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:02.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:03.408 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:03.409 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:03.410 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.411 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.411 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:03.411 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:04.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:05.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.872 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:07.410 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:07.412 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:07.412 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:07.412 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.412 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:07.412 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.412 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:07.413 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.829 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:09.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:11.413 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:11.414 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:11.414 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:11.414 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.415 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:11.416 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:12.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:13.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:15.416 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:15.417 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:15.417 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:15.417 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.417 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:15.417 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.417 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:15.418 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:16.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:17.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:18.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:19.418 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:19.419 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.420 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:19.420 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.420 7f2a39836e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.420 7f2a39836e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:19.420 7f2a39836e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:20.026 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:11:20.027 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:11:20.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:21.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:22.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:23.419 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:23.421 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.422 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:23.422 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.422 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.422 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:23.422 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:24.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:25.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:27.421 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:27.422 7f2a380c6400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:27.422 7f2a380c6400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:27.422 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.422 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:27.422 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.422 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:27.423 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:28.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:29.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:31.423 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:31.424 7f2a380c6400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:31.424 7f2a380c6400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:31.424 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.424 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:31.424 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.424 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:31.425 7f2a380c6400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:33.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:35.425 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:35.426 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:35.426 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:35.427 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.428 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.428 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:35.428 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:36.873 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.830 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:39.427 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:39.428 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:39.429 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.430 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:39.430 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.430 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.430 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:39.430 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:40.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:41.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:43.429 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:43.430 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:43.431 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.432 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:43.432 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.432 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.432 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:43.432 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:44.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:45.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:46.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:47.432 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:47.433 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:47.433 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:47.433 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.433 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:47.433 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.433 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:47.434 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:48.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:49.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:50.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:51.434 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:51.435 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:51.435 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:51.435 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:51.436 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:52.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:53.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:54.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:55.436 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:55.437 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:55.437 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:55.438 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.438 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:55.438 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.438 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:55.438 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:55.438 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.439 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:55.439 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.439 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.439 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:55.439 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:56.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:57.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:58.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:59.438 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.440 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:59.441 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.441 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.441 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:59.441 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:00.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:01.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:03.441 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:03.442 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:03.443 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.443 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:03.443 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.443 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.443 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:03.443 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:04.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:05.873 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.874 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:07.443 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:07.444 7f2a380c69a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:07.445 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.446 7f2a380c69a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.446 7f2a380c69a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:07.446 7f2a380c69a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.831 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:10.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:11.446 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:11.447 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:11.447 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:11.447 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.447 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:11.447 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:11.448 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:12.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:13.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:15.449 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:15.450 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:15.450 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:15.450 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.450 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:15.450 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.450 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:15.451 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:16.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:17.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:18.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:19.450 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:19.452 7f2a38d07400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:19.452 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:19.452 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.452 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:19.452 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.452 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:19.453 7f2a38d07400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.928 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:12:20.929 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:12:21.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:22.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:23.453 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:23.454 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:23.454 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:23.454 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.455 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.456 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:23.456 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:24.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:25.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:26.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:27.455 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:27.456 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:27.457 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.457 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:27.457 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.457 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.457 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:27.457 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:28.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:29.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:31.456 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:31.458 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:31.459 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.459 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:31.459 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.459 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.459 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:31.459 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:32.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:34.872 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:35.459 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:35.460 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:35.460 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:35.460 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.460 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:35.460 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.460 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:35.461 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.874 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:36.875 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.832 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:39.461 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:39.462 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:39.462 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:39.462 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.462 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:39.462 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.462 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:39.463 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.834 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:40.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:41.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:43.463 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:43.464 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.465 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.466 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:43.466 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:44.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:45.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:46.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:47.466 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:47.467 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:47.467 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.468 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.469 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:47.469 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:48.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:49.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:50.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:51.468 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:51.469 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:51.469 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:51.469 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.469 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:51.469 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.469 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:51.470 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:52.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:53.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:54.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:55.471 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:55.472 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:55.472 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:55.472 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.472 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:55.472 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.472 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:55.473 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:56.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:57.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:59.474 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:59.475 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:59.476 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.477 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.477 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:59.477 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:00.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:01.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:02.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:03.476 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:03.478 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:03.478 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:03.478 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:03.479 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.480 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.480 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:03.480 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:04.873 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:05.874 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.875 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:07.479 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:07.480 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:07.480 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:07.480 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.480 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:07.480 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.480 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:07.481 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.833 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:09.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:11.481 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:11.482 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:11.483 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.484 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:11.484 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.484 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.484 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:11.484 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:12.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:13.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:15.484 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:15.485 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.486 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:15.487 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.845 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:16.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:17.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:18.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:19.486 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:19.487 7f2a3994a400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:19.487 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.488 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:19.489 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.489 7f2a3994a400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.489 7f2a3994a400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:19.489 7f2a3994a400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:20.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:22.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:23.489 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:23.490 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:23.490 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:23.490 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.490 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:23.490 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:23.491 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:24.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:25.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:27.491 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:27.492 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:27.492 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:27.492 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.492 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:27.492 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.492 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:27.493 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:28.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:29.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:31.492 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:31.494 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:31.494 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:31.494 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.494 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:31.494 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.494 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:31.495 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:32.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:33.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.872 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:35.494 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:35.496 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:35.496 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:35.496 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.496 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:35.497 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.874 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:36.875 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:39.497 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:39.498 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:39.498 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:39.498 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.498 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:39.499 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:40.838 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:41.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:43.499 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:43.501 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:43.501 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:43.501 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.501 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:43.501 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.501 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:43.502 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.843 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:44.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:45.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:46.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:47.502 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:47.503 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:47.503 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:47.503 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:47.504 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.848 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:48.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:49.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:50.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:51.504 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:51.506 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:51.506 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:51.506 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.506 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:51.506 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.507 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:51.507 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:51.507 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.507 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:51.507 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.507 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.508 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:51.508 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:52.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:53.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:54.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:55.507 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:55.509 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:55.509 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:55.509 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.509 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:55.509 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.509 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:55.510 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:56.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:57.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:59.510 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:59.511 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:59.511 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:59.512 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:00.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:01.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:03.513 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:03.513 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.514 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.515 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:03.515 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:04.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:05.872 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.873 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:07.515 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:07.516 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:07.516 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:07.516 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.516 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:07.516 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.516 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:07.517 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.835 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:09.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:11.517 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:11.519 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:11.519 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:11.519 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.519 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:11.519 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.519 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:11.520 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:11.520 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.520 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:11.520 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.520 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.521 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:11.521 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.840 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:12.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:13.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:15.520 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:15.521 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:15.521 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:15.521 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.521 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:15.521 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.521 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:15.522 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:16.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:17.851 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:18.853 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:19.522 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.524 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:19.525 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.525 7f2a380c6ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.525 7f2a380c6ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:19.525 7f2a380c6ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:20.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:21.857 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:22.510 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:14:22.511 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:14:22.511 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:14:22.511 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:14:22.511 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:14:22.511 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:14:22.511 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:14:22.512 7f2a39836d60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:14:22.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:23.525 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:23.526 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:23.526 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:23.527 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.527 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:23.527 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.527 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:23.527 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:23.527 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.528 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:23.528 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.528 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.528 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:23.528 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.861 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:24.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:25.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.864 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:27.527 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:27.528 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:27.529 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.529 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:27.529 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.529 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.529 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:27.529 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.866 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:28.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:29.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:31.529 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:31.530 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:31.531 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.532 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.532 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:31.532 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:32.873 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:33.875 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.876 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:35.531 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:35.534 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:35.534 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:35.534 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.534 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:35.534 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.534 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:35.535 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.878 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:36.879 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.836 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:39.534 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:39.549 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:39.549 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:39.549 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.549 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.550 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:39.551 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.837 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:40.839 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:41.841 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.842 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:43.550 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:43.551 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:43.551 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.552 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.553 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:43.553 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.844 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:44.846 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:45.847 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:46.849 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:47.552 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.554 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:47.555 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.555 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.555 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:47.555 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.850 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:48.852 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:49.854 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:50.855 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:51.555 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:51.556 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:51.556 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:51.556 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:51.556 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:51.556 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:51.557 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.856 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:52.858 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:53.859 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:54.860 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:55.557 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:55.558 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:55.558 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:55.558 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:55.559 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.862 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:56.863 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:57.865 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:58.867 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:59.560 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:59.561 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.562 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:59.562 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.562 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.562 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:59.562 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.868 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:00.869 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:01.340 7f2a38d07a40 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:01.344 7f2a38d07a40 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:01.870 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:02.871 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:03.562 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:03.563 7f2a38d07a40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:03.563 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:03.563 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:03.563 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:03.563 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:03.563 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691826931937 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.249:8086, 10.0.0.131:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:03.564 7f2a38d07a40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.872 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:04.874 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:05.875 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:06.877 7f2a380eb7c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.