09/01/2025 01:56:34.503 7fc4b84f5040 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:56:34.503 7fc4b84f5040 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:56:34.503 7fc4b84f5040 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:56:34.503 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.249:2181,10.0.0.38:2181,10.0.0.131:2181 09/01/2025 01:56:34.503 7fc4b84f5040 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:56:34.504 7fc4b46cc9a0 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:56:34.504 7fc4b46cc9a0 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:56:34.513 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:56:34.513 7fc4b46ccd60 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794513765 collector_name = collector_ip = collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:56:34.514 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.521 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:34.521 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:56:34.521 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.517 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:38.518 7fc4b46ccd60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:38.518 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:38.518 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794513765 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:38.519 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.521 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:42.522 7fc4b46ccd60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:42.523 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:42.523 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:42.523 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:42.523 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:42.523 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794513765 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:42.524 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.527 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:46.528 7fc4b46ccd60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:46.528 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:46.528 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.529 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:46.530 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794513765 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:46.530 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.529 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:50.530 7fc4b46ccd60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:50.530 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:50.530 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:50.530 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:50.530 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:50.530 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794513765 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:50.531 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.533 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:54.534 7fc4b46ccd60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:54.534 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:54.534 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:54.534 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:54.534 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:54.534 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691794513765 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:56:54.535 7fc4b46ccd60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.704 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.249:9041', '10.0.0.38:9041', '10.0.0.131:9041'] 09/01/2025 01:56:54.704 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:56:54.704 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:56:54.705 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:56:54.705 7fc4b46cce00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.705 7fc4b46cce00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.705 7fc4b46cce00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.709 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:56:54.740 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:56:54.742 7fc4b3bf5360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.742 7fc4b3bf5360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.742 7fc4b3bf5360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.751 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:56:54.751 7fc4b3bf5860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.751 7fc4b3bf5860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:56:54.751 7fc4b3bf5860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.762 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:56:54.798 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:56:54.798 7fc4b4689cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.798 7fc4b4689cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.798 7fc4b4689cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.912 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:56:54.942 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:56:54.943 7fc4b3952540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.943 7fc4b3952540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.943 7fc4b3952540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.943 7fc4b3952540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.943 7fc4b3952540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:54.943 7fc4b3952540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.958 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:56:54.958 7fc4b3952540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:54.958 7fc4b3952540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:54.959 7fc4b3952540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.053 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:55.053 7fc4b3b49860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.054 7fc4b3b49860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:55.054 7fc4b3b49860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.069 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:55.070 7fc4b3b49860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.070 7fc4b3b49860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:55.070 7fc4b3b49860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.072 7fc4b3b49b80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:56:55.073 7fc4b3b49e00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.073 7fc4b3b49e00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:56:55.073 7fc4b3b49e00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.090 7fc4b3b49b80 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:56:55.090 7fc4b3b49ea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.093 7fc4b3b49ea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:55.093 7fc4b3b49ea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.167 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-fabric']: Unknown id: fabric default-global-system-config:default-fabric 09/01/2025 01:56:55.220 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management']: Unknown id: policy_management default-policy-management 09/01/2025 01:56:55.265 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management', 'default-application-policy-set']: Unknown id: application_policy_set default-policy-management:default-application-policy-set 09/01/2025 01:56:55.390 7fc4b84f5040 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric']: Unknown id: virtual_network default-domain:default-project:ip-fabric 09/01/2025 01:57:05.540 7ff0aa38c800 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:57:05.540 7ff0aa38c800 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:57:05.540 7ff0aa38c800 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:57:05.541 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.249:2181,10.0.0.38:2181,10.0.0.131:2181 09/01/2025 01:57:05.541 7ff0aa38c800 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 01:57:05.542 7ff0a5ae9900 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/01/2025 01:57:05.542 7ff0a5ae9900 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/01/2025 01:57:05.542 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:57:05.542 7ff0a5ae9cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/01/2025 01:57:05.543 7ff0a5ae9cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.557 7ff0a5ae9d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.557 7ff0a5ae9d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:05.557 7ff0a5ae9d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.599 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.249:9041', '10.0.0.38:9041', '10.0.0.131:9041'] 09/01/2025 01:57:05.599 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/01/2025 01:57:05.600 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/01/2025 01:57:05.600 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:57:05.600 7ff0a66a8360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.600 7ff0a66a8360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:05.601 7ff0a66a8360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.605 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:57:05.665 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:57:05.667 7ff0a65db5e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.667 7ff0a65db5e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:05.667 7ff0a65db5e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.672 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:57:05.673 7ff0a65db2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.673 7ff0a65db2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:57:05.673 7ff0a65db2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.696 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:57:05.732 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:57:05.733 7ff0a65984a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.733 7ff0a65984a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:05.733 7ff0a65984a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.816 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.249:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.131:5673//'] 09/01/2025 01:57:05.827 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/01/2025 01:57:05.828 7ff0a66a8220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.828 7ff0a66a8220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:05.829 7ff0a66a8220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.829 7ff0a66a8220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.829 7ff0a66a8220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:05.829 7ff0a66a8220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.834 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:57:05.834 7ff0a66a8220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.834 7ff0a66a8220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:05.834 7ff0a66a8220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.934 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:57:05.935 7ff0a5a5e040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.935 7ff0a5a5e040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:05.936 7ff0a5a5e040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.940 7ff0aa38c800 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:57:05.941 7ff0a5a5e040 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.941 7ff0a5a5e040 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:05.941 7ff0a5a5e040 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.942 7ff0a5a5e360 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/01/2025 01:57:05.942 7ff0a5a5e5e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.943 7ff0a5a5e5e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:57:05.943 7ff0a5a5e5e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.947 7ff0a5a5e360 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:57:05.947 7ff0a5a5e860 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:05.947 7ff0a5a5e860 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:57:05.948 7ff0a5a5e860 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.291 7ff0a5a5e2c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.38_0 09/01/2025 01:57:06.291 7ff0a5a5e2c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly) 09/01/2025 01:57:06.299 7ff0a5a5e2c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (nightly) 09/01/2025 01:57:06.305 7ff0a5a5e2c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running walk 09/01/2025 01:57:06.314 7ff0a5a5e2c0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 09/01/2025 01:57:06.479 7ff0a4cfce00 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.479 7ff0a4cfce00 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:06.479 7ff0a4cfce00 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.486 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting consumer greenlet 09/01/2025 01:57:06.486 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:57:06.487 7ff0a4cce220 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting publisher greenlet 09/01/2025 01:57:06.487 7ff0a4cce220 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Publisher connection 09/01/2025 01:57:06.488 7ff0aa38c800 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-2..dummy 09/01/2025 01:57:06.488 7ff0aa38c800 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 01:57:06.489 7ff0a4cfcea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.490 7ff0a4cfcea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ConfigApiWorkerTrace: data = << name = worker-0 worker_id = 0 introspect_port = 8084 admin_port = 8095 introspect_url = http://10.0.0.38:8084 >> 09/01/2025 01:57:06.490 7ff0a4cfcea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.494 7ff0a5a5e2c0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Time elapsed in resyncing db: 0:00:00.202913 09/01/2025 01:57:06.495 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-a6d570db-61ed-483e-9d29-23c0445f8de7', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:57:06.496 7ff0a4cce0e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:57:06.499 7ff0a4cce220 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 09/01/2025 01:57:06.501 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/01/2025 01:57:06.503 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-4817b85a-0381-44d9-98b1-b4cc92223331', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:57:06.504 7ff0a4cceae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.504 7ff0a4cceae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:57:06.086429", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:57:06.505 7ff0a4cceae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.509 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'config_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:57:06.307811', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:06.307811', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13128683577167256048, 'uuid_mslong': 10033920323497510113}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8b3fa55f-4aa4-44e1-b632-7427998f19f0'}, 'oper': 'CREATE', 'request-id': 'req-b2c1949a-b9e4-4de8-b1f2-23c0b93c352f', 'type': 'config_node', 'uuid': '8b3fa55f-4aa4-44e1-b632-7427998f19f0'} 09/01/2025 01:57:06.509 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:06.510 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-178dcc85-7d0f-4b7f-a81d-367ea0808919', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:57:06.512 7ff0a4ccec20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.512 7ff0a4ccec20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { config_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10033920323497510113, "uuid_lslong": 13128683577167256048}, "enable": true, "created": "2025-09-01T01:57:06.307811", "last_modified": "2025-09-01T01:57:06.307811", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "8b3fa55f-4aa4-44e1-b632-7427998f19f0" } deleted = False >> 09/01/2025 01:57:06.512 7ff0a4ccec20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.515 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/01/2025 01:57:06.516 7ff0a4ccec20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.516 7ff0a4ccec20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:57:06.330561", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:57:06.516 7ff0a4ccec20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:06.537 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-2..dummy'] 09/01/2025 01:57:06.553 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 01:57:06.558 7ff0a4cce0e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/01/2025 01:57:06.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-3995-2..dummy'] 09/01/2025 01:57:07.346 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-627a4ee1-7b6b-4f96-97ad-6e4314a60029', 'type': 'application_policy_set', 'uuid': 'fe91bbbb-319d-408f-a286-f6683c9cd494'} 09/01/2025 01:57:07.356 7ff0a4cce7c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.356 7ff0a4cce7c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "75c808b9-5f11-4499-9a9b-13d38152369a"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18343649169922408591, "uuid_lslong": 11711318808577496212}, "enable": true, "created": "2025-09-01T01:56:55.332205", "last_modified": "2025-09-01T01:57:07.333259", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "b3ca7d55-fca8-4f51-ad6c-b6601d656a7e"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "fe91bbbb-319d-408f-a286-f6683c9cd494" } deleted = False >> 09/01/2025 01:57:07.356 7ff0a4cce7c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.535 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-d73327dd-fcdc-48c2-9ec5-f2dafab03c46', 'type': 'network_ipam', 'uuid': '4c97a050-8ab3-4af5-92d5-479e80a83da1'} 09/01/2025 01:57:07.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:07.744 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-34de5f60-0dc2-44ba-9d39-a1c3bc2a7184', 'type': 'project', 'uuid': '2c2387c7-92b5-4a0f-b4e2-008f72bcc90a'} 09/01/2025 01:57:07.749 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.749 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3180535053083298319, "uuid_lslong": 13033980887668934922}, "enable": true, "created": "2025-09-01T01:56:55.060606", "last_modified": "2025-09-01T01:57:07.735619", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "4bb7dde1-472f-41f0-9da0-5eb4249cd377"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "2c2387c7-92b5-4a0f-b4e2-008f72bcc90a" } deleted = False >> 09/01/2025 01:57:07.749 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.596 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'config_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:57:08.589359', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:08.589359', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9854752206779726768, 'uuid_mslong': 3885232349240774114}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0'}, 'oper': 'CREATE', 'request-id': 'req-080b3d52-9fcf-4ddd-853b-6402086b7f46', 'type': 'config_node', 'uuid': '35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0'} 09/01/2025 01:57:08.597 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:08.597 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.598 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { config_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3885232349240774114, "uuid_lslong": 9854752206779726768}, "enable": true, "created": "2025-09-01T01:57:08.589359", "last_modified": "2025-09-01T01:57:08.589359", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "35eb1e3c-9e5a-4de2-88c3-1ceb5a9a3bb0" } deleted = False >> 09/01/2025 01:57:08.598 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:08.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:09.543 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:09.545 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:09.545 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:09.545 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:09.545 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:09.545 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:09.545 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:09.546 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:09.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:10.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:11.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:12.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:13.545 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:13.546 7ff0a5a5e2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:13.546 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:13.546 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:13.547 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:13.547 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:13.547 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:13.547 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:13.547 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.548 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:13.548 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.548 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.548 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:13.548 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:13.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:14.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:15.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:16.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:17.547 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:17.548 7ff0a67254a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:17.548 7ff0a67254a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:17.548 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:17.549 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:17.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:18.450 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'config_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:57:18.438111', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:18.438111', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11481679389149772070, 'uuid_mslong': 9952658429463971103}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8a1ef21c-32ff-451f-9f57-1e9267323926'}, 'oper': 'CREATE', 'request-id': 'req-29c41265-556e-4193-aa1b-64401b965cb7', 'type': 'config_node', 'uuid': '8a1ef21c-32ff-451f-9f57-1e9267323926'} 09/01/2025 01:57:18.451 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 09/01/2025 01:57:18.451 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.452 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { config_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9952658429463971103, "uuid_lslong": 11481679389149772070}, "enable": true, "created": "2025-09-01T01:57:18.438111", "last_modified": "2025-09-01T01:57:18.438111", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "8a1ef21c-32ff-451f-9f57-1e9267323926" } deleted = False >> 09/01/2025 01:57:18.452 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:18.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:19.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:20.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:21.549 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:21.550 7ff0a5a5e2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:21.550 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:21.550 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:21.550 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:21.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:21.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:22.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:23.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:24.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:25.551 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:25.553 7ff0a5a5e2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:25.553 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:25.553 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:25.553 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:25.553 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:25.553 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:25.554 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:25.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:26.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:27.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:28.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:29.554 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:29.556 7ff0a5a5e2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:29.556 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:29.556 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.557 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.558 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:29.558 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:29.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:30.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:31.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:32.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:33.557 7ff0a66a8180 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:33.558 7ff0a66a8180 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:33.558 7ff0a66a8180 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:33.558 7ff0a66a8180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:33.558 7ff0a66a8180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:33.558 7ff0a66a8180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:33.558 7ff0a66a8180 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:33.559 7ff0a66a8180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:33.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:34.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:35.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:37.559 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:37.560 7ff0a4dc2220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:37.561 7ff0a4dc2220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:37.561 7ff0a4dc2220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:37.561 7ff0a4dc2220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:37.561 7ff0a4dc2220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:37.561 7ff0a4dc2220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:37.561 7ff0a4dc2220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:37.562 7ff0a4dc2220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:38.561 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:39.563 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:40.565 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:41.561 7ff0a4dc2ae0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:41.563 7ff0a4dc2ae0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:41.564 7ff0a4dc2ae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.564 7ff0a4dc2ae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/01/2025 01:57:41.564 7ff0a4dc2ae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.564 7ff0a4dc2ae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.564 7ff0a4dc2ae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:41.564 7ff0a4dc2ae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:41.566 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:42.568 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:43.398 7ff0a5a5e2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.398 7ff0a5a5e2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:43.399 7ff0a5a5e2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.423 7ff0a4cce720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:ingress-access-control-list 09/01/2025 01:57:43.432 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = http://10.0.0.38:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "ingress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ingress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.470 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:43.460769', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.460769', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11925350455766867329, 'uuid_mslong': 15648137040972304299}}, 'parent_type': 'security-group', 'parent_uuid': '62ca173b-4211-434e-8067-4fcb628e2505', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'}, 'oper': 'CREATE', 'request-id': 'req-e9e6110b-15a8-45e1-b6ff-9bbe5f1606d8', 'type': 'access_control_list', 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'} 09/01/2025 01:57:43.519 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:57:43.512816', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.512816', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11154437056324030764, 'uuid_mslong': 14414019718996774673}}, 'parent_type': 'security-group', 'parent_uuid': '62ca173b-4211-434e-8067-4fcb628e2505', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'}, 'oper': 'CREATE', 'request-id': 'req-a4d2e73b-68f2-4c38-a76b-9509eebec0d9', 'type': 'access_control_list', 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'} 09/01/2025 01:57:43.569 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:43.603 7ff0a4cce720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000000'] not found in DB: Unknown id: route_target target:64512:8000000 09/01/2025 01:57:43.603 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000000 url = http://10.0.0.38:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000000"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000000"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.617 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-09-01T01:57:43.610873', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.610873', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13092489608894395420, 'uuid_mslong': 18272932050593923086}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'fd967ee2-c880-400e-b5b1-ddeeaba4b41c'}, 'oper': 'CREATE', 'request-id': 'req-667bdd9c-a819-416b-a362-7c8709dec24e', 'type': 'route_target', 'uuid': 'fd967ee2-c880-400e-b5b1-ddeeaba4b41c'} 09/01/2025 01:57:43.645 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-d38d9b73-6e23-420e-ad61-26940b0d8417', 'type': 'routing_instance', 'uuid': '26898347-c171-4c1a-8697-64f6fe425338'} 09/01/2025 01:57:43.689 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-09-01T01:57:43.671641', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.671641', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13570455496092117547, 'uuid_mslong': 12041940091083179522}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'a71d90b6-3f8c-4602-bc53-f15fd4d16a2b'}, 'oper': 'CREATE', 'request-id': 'req-848a89e9-1bb6-4095-a058-f8b3aa42453b', 'type': 'route_target', 'uuid': 'a71d90b6-3f8c-4602-bc53-f15fd4d16a2b'} 09/01/2025 01:57:43.697 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = b402f478-6c09-4cce-978c-bf08921c9092 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:dci-network url = http://10.0.0.38:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "routing-instance", "uuid": "b402f478-6c09-4cce-978c-bf08921c9092", "ref-type": "route-target", "ref-uuid": "a71d90b6-3f8c-4602-bc53-f15fd4d16a2b", "ref-fq-name": ["target:64512:8000001"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.711 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-7fd48cc7-6227-4e3e-9208-830d1b311a69', 'type': 'routing_instance', 'uuid': 'b402f478-6c09-4cce-978c-bf08921c9092'} 09/01/2025 01:57:43.746 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-09-01T01:57:43.738435', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.738435', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13504741517806881909, 'uuid_mslong': 10871219233836909678}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': '96de544b-843e-446e-bb6a-7adbf8594c75'}, 'oper': 'CREATE', 'request-id': 'req-eafc60a5-2b5d-4bf9-8cf2-e091b0ef8565', 'type': 'route_target', 'uuid': '96de544b-843e-446e-bb6a-7adbf8594c75'} 09/01/2025 01:57:43.771 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-dd79017c-e717-4d14-ba03-dacf3e729740', 'type': 'routing_instance', 'uuid': 'b092f404-1958-4858-b56e-1b2fb3b6dcd2'} 09/01/2025 01:57:43.796 7ff0a4cce720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000003'] not found in DB: Unknown id: route_target target:64512:8000003 09/01/2025 01:57:43.797 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000003 url = http://10.0.0.38:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000003"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "20dbadfeabfd403fb6d2794acee4de16", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000003"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.814 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-09-01T01:57:43.808184', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.808184', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10211255544152584447, 'uuid_mslong': 14240605882422477187}}, 'perms2': {'global_access': 0, 'owner': '20dbadfeabfd403fb6d2794acee4de16', 'owner_access': 7, 'share': []}, 'uuid': 'c5a0cb99-a021-4183-8db5-aad0dbf3f0ff'}, 'oper': 'CREATE', 'request-id': 'req-e8279a5b-a4e6-4fee-aa01-864a568ee38c', 'type': 'route_target', 'uuid': 'c5a0cb99-a021-4183-8db5-aad0dbf3f0ff'} 09/01/2025 01:57:43.839 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-94fd4bc4-4c44-489d-8a70-3b0e401279f1', 'type': 'routing_instance', 'uuid': 'fa187b15-223c-4e06-9490-9a6decc1dbc7'} 09/01/2025 01:57:43.886 7ff0a4cce720 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'] not found in DB: Unknown id: access_control_list default-domain:default-project:ip-fabric:ip-fabric 09/01/2025 01:57:43.892 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.38:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "parent_type": "virtual-network", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ip-fabric"}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:43.933 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-01T01:57:43.914671', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:43.914671', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12434635487439495936, 'uuid_mslong': 17013906886330174589}}, 'parent_type': 'virtual-network', 'parent_uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'}, 'oper': 'CREATE', 'request-id': 'req-7ee2abe6-6195-4e08-b9e2-a1db250d1fde', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 01:57:43.954 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-dacc4d4b-9503-4813-aead-82a4bd87ba77', 'type': 'access_control_list', 'uuid': 'd92959bb-3bc9-4fab-a57f-5b0e6a06dd81'} 09/01/2025 01:57:43.973 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-28a2300c-ad5b-4055-b6ef-fd6c1340a532', 'type': 'access_control_list', 'uuid': 'c808e290-d047-4f11-9acc-85676ba5d12c'} 09/01/2025 01:57:44.571 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:45.563 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:45.564 7ff0a5ae9cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:45.564 7ff0a5ae9cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:45.564 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.565 7ff0a5ae9cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.566 7ff0a5ae9cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:45.566 7ff0a5ae9cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:46.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:47.330 7ff0a4cceae0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 09/01/2025 01:57:47.330 7ff0a4cceae0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 09/01/2025 01:57:47.330 7ff0a4cceae0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 09/01/2025 01:57:47.437 7ff0a4cceae0 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 75c808b9-5f11-4499-9a9b-13d38152369a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = http://10.0.0.38:8082/global-vrouter-config/75c808b9-5f11-4499-9a9b-13d38152369a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/opt/contrail/utils/provision_linklocal.py remote_ip = 10.0.0.131 body = {"global-vrouter-config": {"uuid": "75c808b9-5f11-4499-9a9b-13d38152369a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "linklocal_services": {"linklocal_service_entry": [{"linklocal_service_name": "metadata", "linklocal_service_ip": "169.254.169.254", "linklocal_service_port": 80, "ip_fabric_DNS_service_name": "", "ip_fabric_service_port": 8775, "ip_fabric_service_ip": ["10.0.0.249"]}]}}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:47.460 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-b9cee00c-1b45-404e-af54-eadf4c384e92', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:47.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:47.689 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-bf1780e4-3d48-4dbf-97d0-5077fe0b1810', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:47.833 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-f41da2eb-8d35-4dcf-a758-8cf2d89040a0', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:48.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:49.091 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.25', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.25', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T01:57:49.074180', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.074180', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9621979915837513096, 'uuid_mslong': 12350586173020262281}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'}, 'oper': 'CREATE', 'request-id': 'req-c0a3b8a3-b9c0-47b4-aa6d-0410935f0b7e', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:49.092 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.093 7ff0a4cced60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.093 7ff0a4cced60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.074180", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:49.093 7ff0a4cced60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.365 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-bbc43391-f503-4e5f-ae26-aa04ed6e08c3', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:49.565 7ff0a4cced60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:49.567 7ff0a4cced60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:49.567 7ff0a4cced60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:49.567 7ff0a4cced60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:49.567 7ff0a4cced60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:49.567 7ff0a4cced60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:49.567 7ff0a4cced60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:49.568 7ff0a4cced60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:49.568 7ff0a4cced60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.568 7ff0a4cced60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:49.569 7ff0a4cced60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.569 7ff0a4cced60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.569 7ff0a4cced60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:49.569 7ff0a4cced60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:49.719 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-01T01:57:49.704740', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.704740', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13465058781904455200, 'uuid_mslong': 269198390335917880}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-7d26a5a0-84f7-42ec-800e-7cdc9b85f184', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.768 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.129', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.129', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'id_perms': {'created': '2025-09-01T01:57:49.750291', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.750291', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11857703135272596087, 'uuid_mslong': 7997815229011673419}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'}, 'oper': 'CREATE', 'request-id': 'req-8f2e7ba5-2d05-4723-9303-eb4f0aa1e337', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:49.768 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.769 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.769 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:49.750291", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:49.769 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.801 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.193', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.193', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T01:57:49.778205', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:57:49.778205', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10075118084394099490, 'uuid_mslong': 1863496716102945915}}, 'parent_type': 'routing-instance', 'parent_uuid': '24dce67f-6d24-4317-b7b6-230ab5b033b7', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'}, 'oper': 'CREATE', 'request-id': 'req-e5b882a7-327b-41ce-b42a-1a0c876931dc', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:49.801 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/01/2025 01:57:49.802 7ff0a4cceea0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.802 7ff0a4cceea0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:49.778205", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:49.802 7ff0a4cceea0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.823 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 6efdf293-9873-414b-a48f-062d3df57277 object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-3995-3 url = http://10.0.0.38:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"type": "bgp-router", "uuid": "6efdf293-9873-414b-a48f-062d3df57277", "ref-type": "bgp-router", "ref-uuid": null, "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "operation": "ADD", "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}} domain = default-domain project = admin user = admin >> 09/01/2025 01:57:49.841 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-02b807bd-14fd-4f1e-b82a-c25261b53d43', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.844 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-01T01:57:49.827148', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.827148', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9281412384941033885, 'uuid_mslong': 18032248181005373295}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4d39c21d-ea49-4fbd-881e-a4e63a6130e5', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:49.854 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-eaedbf1a-4d99-4ee5-a020-a5df73fc9289', 'type': 'alarm', 'uuid': '03bc6282-7e14-4f38-badd-7f9f072b4620'} 09/01/2025 01:57:49.889 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'oper': 'UPDATE', 'request-id': 'req-3b0d910e-4eff-4ddc-9b7a-42f2a96d0b1a', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:49.893 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.893 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:49.862739", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:49.893 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.894 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-3b0d910e-4eff-4ddc-9b7a-42f2a96d0b1a', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:49.902 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:49.902 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.859846", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:49.902 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.012 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'oper': 'UPDATE', 'request-id': 'req-8e0e1c75-cdee-4b81-a97d-47313a4410a5', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:50.018 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.018 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:49.990399", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:50.018 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.018 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-8e0e1c75-cdee-4b81-a97d-47313a4410a5', 'type': 'bgp_router', 'uuid': 'ab6618b0-2ab9-4f89-8588-23c107701d88'} 09/01/2025 01:57:50.039 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-01T01:57:49.990142', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:49.990142', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13643264217144023489, 'uuid_mslong': 14422323711782898249}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-d198a7f7-9028-417a-81b9-278fb7ce7f83', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.040 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.040 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.25", "address": "10.20.0.25", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12350586173020262281, "uuid_lslong": 9621979915837513096}, "enable": true, "created": "2025-09-01T01:57:49.074180", "last_modified": "2025-09-01T01:57:49.981032", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab6618b0-2ab9-4f89-8588-23c107701d88" } deleted = False >> 09/01/2025 01:57:50.040 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.075 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-64ef5a1c-639c-4c3e-b6bf-c4d98fa5bb18', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:50.084 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-6248bd5b-43c4-48ea-a023-a61144970ae3', 'type': 'alarm', 'uuid': 'fa3f6a2e-8282-4f6f-80ce-33639a52499d'} 09/01/2025 01:57:50.099 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-948d4f9f-cccb-41f5-b615-4fdb2635b0d4', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:50.107 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-2'], 'oper': 'UPDATE', 'request-id': 'req-6f660d61-332b-4f6d-adba-f26f42913c20', 'type': 'bgp_router', 'uuid': '19dc7863-db6c-4c7b-8bd2-02847a6ca322'} 09/01/2025 01:57:50.119 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-3995-3'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-6f660d61-332b-4f6d-adba-f26f42913c20', 'type': 'bgp_router', 'uuid': '6efdf293-9873-414b-a48f-062d3df57277'} 09/01/2025 01:57:50.128 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.129 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.193", "address": "10.20.0.193", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "6efdf293-9873-414b-a48f-062d3df57277"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1863496716102945915, "uuid_lslong": 10075118084394099490}, "enable": true, "created": "2025-09-01T01:57:49.778205", "last_modified": "2025-09-01T01:57:50.091698", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "19dc7863-db6c-4c7b-8bd2-02847a6ca322" } deleted = False >> 09/01/2025 01:57:50.129 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.140 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-45aae796-5805-4664-8f68-8e9977d1075b', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:50.141 7ff0a67254a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.141 7ff0a67254a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.129", "address": "10.20.0.129", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "19dc7863-db6c-4c7b-8bd2-02847a6ca322"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ab6618b0-2ab9-4f89-8588-23c107701d88"}]display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3995-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7997815229011673419, "uuid_lslong": 11857703135272596087}, "enable": true, "created": "2025-09-01T01:57:49.750291", "last_modified": "2025-09-01T01:57:50.085318", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "24dce67f-6d24-4317-b7b6-230ab5b033b7"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6efdf293-9873-414b-a48f-062d3df57277" } deleted = False >> 09/01/2025 01:57:50.141 7ff0a67254a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:50.162 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.146892', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.146892', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12719190076713101290, 'uuid_mslong': 9355486960643295514}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-ab3956ca-bc75-4fc4-90b9-5fb4002eab77', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.260 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-01T01:57:50.244269', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.244269', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12345975188693128422, 'uuid_mslong': 12789451347087803108}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3ebf1ad0-4d29-4eb6-b68c-83cb5e0c2c57', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.278 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-4644b680-219f-41ac-93f0-2af58d6290f6', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.284 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-a1db2026-671d-43f2-beab-63ca7fd085d9', 'type': 'alarm', 'uuid': 'c8266300-cc83-4a49-bd56-9c8537aad9c1'} 09/01/2025 01:57:50.361 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-01T01:57:50.343229', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.343229', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410398409968651569, 'uuid_mslong': 348915920711273427}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-b3df3838-b993-445d-b342-9bdc65b9c290', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.454 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b46a4069-d0ac-41da-baab-6487258bee1f', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.455 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0ea88e73-e57b-474e-b375-5282426f050e', 'type': 'alarm', 'uuid': '81d55dd2-bdf1-491a-b083-a3fc512867ea'} 09/01/2025 01:57:50.464 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-01T01:57:50.437138', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.437138', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10407682553178176690, 'uuid_mslong': 1486187007752490893}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-8cd87a25-7ae6-44da-ac17-b27b16f998eb', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.555 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-01T01:57:50.537284', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.537284', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12650030438488176505, 'uuid_mslong': 7230954819262302855}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-aa98e6eb-63fe-4a1e-abf9-ce5a05ca5b01', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:50.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:50.628 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-a927b894-3265-4f46-a249-ce671ebe63da', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.631 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-939925e0-8ddf-41ea-8f96-f1b4fbe1e932', 'type': 'alarm', 'uuid': 'b17d423d-1f4f-42e4-ab55-b705b7d714e6'} 09/01/2025 01:57:50.648 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-01T01:57:50.636982', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.636982', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11945006111573889988, 'uuid_mslong': 15521484146632968236}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-772d7b53-1bf5-4336-9bd0-cded7ae36250', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:50.726 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.713917', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.713917', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13316427422065746893, 'uuid_mslong': 7023261606246303666}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-17a28f51-9acd-4cc4-9846-b019f65f92ca', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:50.792 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-95b9b3ba-1a0e-4ea3-9245-25030d1a7b8c', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.796 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-ff7124e2-b2c7-4010-879e-a96f7b647965', 'type': 'alarm', 'uuid': '04d7992e-6679-4bd3-8298-737ed5150d31'} 09/01/2025 01:57:50.810 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-01T01:57:50.802208', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-01T01:57:50.802208', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10394145945911723784, 'uuid_mslong': 9299310539720442798}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-4f2c310d-2af7-42f0-b0cc-347196801623', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:50.895 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.885227', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:50.885227', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11080958518291472377, 'uuid_mslong': 3112223694598000309}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-c2925086-dcce-45f5-ab84-4caad16fa4c3', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:50.969 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-6d2502b7-dea2-4907-81a1-e4b8a763d3ee', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.974 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-0b329c1d-2db3-448f-97e9-07c801904437', 'type': 'alarm', 'uuid': '149fff35-9ae3-4f8d-906f-8423cd373cb2'} 09/01/2025 01:57:50.992 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:50.981229', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:57:50.981229', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10109986246743945171, 'uuid_mslong': 10619123826018765353}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-089282b9-7dc7-43b2-a6ef-7aaed496978e', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:51.074 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-01T01:57:51.062109', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.062109', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11650388398251038770, 'uuid_mslong': 10237949505899020380}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-ddf9ec54-1868-42ef-b16e-0e94d2b51aad', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:51.136 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-40c61117-53d3-4682-b6aa-ebab239dc2f6', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:51.140 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-372e3cc2-6865-4786-b23f-a0ea8df70b4e', 'type': 'alarm', 'uuid': '64598318-cd07-4a87-af8d-efa93bf77779'} 09/01/2025 01:57:51.157 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-01T01:57:51.143961', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.143961', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11278860563117070971, 'uuid_mslong': 11803345335316596501}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-1a9eb67e-f428-4a37-96a9-72466ae74d61', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:51.238 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-01T01:57:51.226178', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.226178', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11143410889604691103, 'uuid_mslong': 14282259945798913346}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-32967334-05f5-451f-82ec-3c34074acc51', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:51.301 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-b459fe8b-724f-444f-8092-ca948a7c4bb7', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:51.316 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-c1fbc6a9-f5aa-4b42-83a9-874da7552f7a', 'type': 'alarm', 'uuid': 'd7676398-52ea-482c-a5c5-2fc5683b53c4'} 09/01/2025 01:57:51.330 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-01T01:57:51.312606', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-01T01:57:51.312606', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13487801257560256231, 'uuid_mslong': 10235680329170243091}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-3c23439e-0ccb-4d2b-b98a-24cb37aaab0c', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:51.409 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-01T01:57:51.398898', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.398898', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10710412090359268505, 'uuid_mslong': 3024502048848363658}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-da6c1f03-ff69-4dc6-a1ce-b0a85f4b848d', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:51.476 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-86a74659-a45f-42ec-840b-7a8d9d5f0e2f', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:51.485 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-ff070e25-e45d-4fe3-be1d-003d05e1b568', 'type': 'alarm', 'uuid': '6177a338-54f3-47b2-b8cd-742fd97fe3cd'} 09/01/2025 01:57:51.503 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-01T01:57:51.495240', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.495240', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12799166691106454208, 'uuid_mslong': 15027170893856260812}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-94e1d593-b6f7-40fa-a257-80aad936d295', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:51.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:51.587 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-01T01:57:51.577756', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.577756', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10722918137435541080, 'uuid_mslong': 17809913453040846757}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-220998bd-94b4-4899-848b-14a09e1c3183', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:51.647 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-1b0fb202-a15f-4578-a079-413e846b84b2', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:51.658 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-5ce20252-7ee3-433a-afd4-97a83e0e1942', 'type': 'alarm', 'uuid': '810dc9aa-8015-43ae-903f-6caad1a28708'} 09/01/2025 01:57:51.676 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:51.666562', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.666562', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10608329397487415044, 'uuid_mslong': 12641009152992233842}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-043efcd0-fec7-4d8c-b54c-f5c2f0bffb7e', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:51.765 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:57:51.751034', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.751034', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12665274423476574180, 'uuid_mslong': 14559706905692358710}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-2b1fdcaf-0532-49a7-89d0-ff8f0b532112', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:51.819 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-cceef24b-02aa-4ba9-8e77-159d86d7f996', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:51.831 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-9e20dffb-8801-43ec-8834-42aff3338aa0', 'type': 'alarm', 'uuid': '2b30d6f6-0211-4eb5-99c7-79101d9a1ff9'} 09/01/2025 01:57:51.861 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-01T01:57:51.847772', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-01T01:57:51.847772', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11156140797610135132, 'uuid_mslong': 437379043040970971}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-6e9b4c17-ad10-4551-8af6-958c601596a7', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:52.006 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0ce38428-da3d-46d3-af77-6621d179b916', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:52.039 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-4a3dbbdb-8ea8-4bb0-b802-e55592a31842', 'type': 'alarm', 'uuid': '935eb4db-73bc-4a29-8c4d-e2ed2941efd3'} 09/01/2025 01:57:52.202 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-903126b7-3e5d-4c0d-a96c-0112009b6b6b', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:52.227 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-574ad662-aa66-4387-b6f3-ae5ecedfb9c0', 'type': 'alarm', 'uuid': '8e1480d5-277f-405c-a1ae-7e859515d832'} 09/01/2025 01:57:52.415 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-65485163-b46b-4b7d-b4ea-ac3e45e48b1b', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:52.418 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-4053dbbf-3e1e-47e0-bc3a-9121239cf4a1', 'type': 'alarm', 'uuid': 'a3cde80b-b0cf-4715-9c86-8fec97c15a7b'} 09/01/2025 01:57:52.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:52.611 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b5885dd0-54b6-4de9-b985-e5729ac70731', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:52.644 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-c98a9e53-e0a4-47ea-a8a2-f875f5abdca0', 'type': 'alarm', 'uuid': 'c634c7c0-0322-4d42-9aa5-5929ed1b749f'} 09/01/2025 01:57:52.796 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-be77af09-c6a1-4809-9837-02e4960fbc0f', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:52.807 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-8a8b9aea-1dd9-49ef-a3fa-731e869e0879', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:52.815 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-f824c559-dd9e-47f6-b43c-74bdb695c193', 'type': 'alarm', 'uuid': '8e0c7107-46a5-4e13-bb2e-4bc845f22ee7'} 09/01/2025 01:57:52.982 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-5978a90a-a2bb-4e72-b492-04055837bab6', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:52.985 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-cc861575-b1f8-4b01-8968-321ce8a165f7', 'type': 'alarm', 'uuid': '29f93096-c40c-408a-94a3-070ccbe9c499'} 09/01/2025 01:57:53.149 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-8fd3efd9-9811-4d74-9c37-1d6a2f9d9316', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:53.164 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-d9273f8f-b94b-4a39-bdac-d445fbe24224', 'type': 'alarm', 'uuid': 'd08b3c4f-8364-42cc-b19f-c64aec697ac0'} 09/01/2025 01:57:53.337 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-ea9f552e-98eb-4ef4-80bc-5642d6297bf3', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:53.344 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-e3763da9-84a2-42ee-81cf-a733fdda7b1f', 'type': 'alarm', 'uuid': 'f72985ec-b0f3-47a5-94cf-753bc9704e58'} 09/01/2025 01:57:53.529 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-00c0f81c-27b8-4bf7-b45d-8c60f682ff4c', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:53.532 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-61c5b9c6-20c7-4119-a5a5-c3f28a485fa3', 'type': 'alarm', 'uuid': 'af6de2d9-8690-4972-9338-5b61b5922f04'} 09/01/2025 01:57:53.535 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-fc63db98-24bf-4e4b-a4ef-4f72524f73db', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:53.567 7ff0a4cceae0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:53.569 7ff0a4cceae0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:53.569 7ff0a4cceae0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:53.569 7ff0a4cceae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:53.570 7ff0a4cceae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:53.570 7ff0a4cceae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:53.570 7ff0a4cceae0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:53.570 7ff0a4cceae0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:53.570 7ff0a4cceae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.570 7ff0a4cceae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:53.571 7ff0a4cceae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.571 7ff0a4cceae0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.571 7ff0a4cceae0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:53.571 7ff0a4cceae0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:53.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:53.688 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-3c1aef7d-30c2-406e-983f-8730edafccde', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:53.695 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-e4842649-b912-4285-b75d-22e7fdce5d46', 'type': 'alarm', 'uuid': 'ca0e784a-4398-4836-afc4-17fc122957e4'} 09/01/2025 01:57:53.865 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-257ec96c-0a05-47d5-8ed5-bceba5186da5', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:53.880 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-e087c260-1b7a-4195-aa63-9353258567c6', 'type': 'alarm', 'uuid': '0611e1ec-1ca8-48db-9ad2-92f299cb8e5c'} 09/01/2025 01:57:54.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:54.879 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-02593c6e-2575-40b8-a56f-63f0b1fa86fb', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:54.950 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-9a1cd17d-a4e0-4fed-a459-be82136524dd', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:54.954 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-4438a8df-7099-48db-8441-b15938d14eed', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:55.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:56.229 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-425b5a83-c87b-46df-8e2c-db3c3f271101', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:56.273 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-6c0bdafe-bb88-447a-b0d8-8cbade3c0907', 'type': 'global_vrouter_config', 'uuid': '75c808b9-5f11-4499-9a9b-13d38152369a'} 09/01/2025 01:57:56.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:57.471 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-f00a55d7-ce9e-43e2-9395-3dd8285232ff', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:57.533 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-3b996c4a-3334-4609-ac9b-97b4439f5e60', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 01:57:57.569 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:57.571 7ff0a4ccef40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:57.571 7ff0a4ccef40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:57.571 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:57.571 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:57.571 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:57.571 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:57:57.572 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:57.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:58.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:57:59.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:00.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:01.573 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:01.574 7ff0a4ccef40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:01.574 7ff0a4ccef40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:01.574 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:01.574 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:01.574 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:01.574 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:01.575 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:01.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:02.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:03.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:04.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:05.575 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:05.577 7ff0a4ccef40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:05.577 7ff0a4ccef40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:05.577 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:05.577 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:05.577 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:05.578 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:05.578 7ff0a4ccef40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:05.578 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.578 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:05.578 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.578 7ff0a4ccef40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.579 7ff0a4ccef40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:05.579 7ff0a4ccef40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:05.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:07.561 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:08.562 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.564 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:09.578 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:09.579 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:09.580 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.580 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:09.580 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.580 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:09.580 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:09.580 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:10.566 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:11.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:12.568 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.569 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:13.580 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:13.581 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.582 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:13.582 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.582 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:13.582 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:13.582 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:14.571 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:15.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:16.574 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:17.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:17.581 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:17.583 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:17.584 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.584 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:17.584 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.584 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:17.584 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:17.584 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:18.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:19.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:20.278 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'database_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:58:20.262077', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.262077', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11796017848418923458, 'uuid_mslong': 4531286052305127849}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3ee25ca8-2354-49a9-a3b3-dfbc332d73c2'}, 'oper': 'CREATE', 'request-id': 'req-89536832-5ade-4091-a3e7-dc5b51518fac', 'type': 'database_node', 'uuid': '3ee25ca8-2354-49a9-a3b3-dfbc332d73c2'} 09/01/2025 01:58:20.279 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.279 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.279 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { database_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4531286052305127849, "uuid_lslong": 11796017848418923458}, "enable": true, "created": "2025-09-01T01:58:20.262077", "last_modified": "2025-09-01T01:58:20.262077", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "3ee25ca8-2354-49a9-a3b3-dfbc332d73c2" } deleted = False >> 09/01/2025 01:58:20.279 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.290 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'database_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:58:20.280218', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.280218', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10071480984782399945, 'uuid_mslong': 378361382186796020}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '054035ac-25f8-43f4-8bc5-16983c0e69c9'}, 'oper': 'CREATE', 'request-id': 'req-1e468cdd-34f7-445d-b18b-bd3cca7149c2', 'type': 'database_node', 'uuid': '054035ac-25f8-43f4-8bc5-16983c0e69c9'} 09/01/2025 01:58:20.291 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.291 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.292 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { database_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 378361382186796020, "uuid_lslong": 10071480984782399945}, "enable": true, "created": "2025-09-01T01:58:20.280218", "last_modified": "2025-09-01T01:58:20.280218", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "054035ac-25f8-43f4-8bc5-16983c0e69c9" } deleted = False >> 09/01/2025 01:58:20.292 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.408 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:58:20.395054', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:20.395054', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12835641244198905252, 'uuid_mslong': 4848350493246179075}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4348cd16-0087-4703-b221-5bb498d425a4'}, 'oper': 'CREATE', 'request-id': 'req-67f78842-b75a-435a-8bb4-1eea81bd8c1f', 'type': 'database_node', 'uuid': '4348cd16-0087-4703-b221-5bb498d425a4'} 09/01/2025 01:58:20.409 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/01/2025 01:58:20.409 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.410 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { database_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4848350493246179075, "uuid_lslong": 12835641244198905252}, "enable": true, "created": "2025-09-01T01:58:20.395054", "last_modified": "2025-09-01T01:58:20.395054", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "4348cd16-0087-4703-b221-5bb498d425a4" } deleted = False >> 09/01/2025 01:58:20.410 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:20.579 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:21.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:21.583 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:21.585 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:21.585 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:21.585 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:21.585 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:21.586 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:22.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:23.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:24.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:25.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:25.587 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:25.588 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.589 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:25.589 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.589 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.589 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:25.589 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:26.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:27.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:28.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:29.589 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:29.590 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:29.591 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.591 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:29.591 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.591 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.591 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:29.591 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:29.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:30.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:31.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:32.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:33.591 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:33.592 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:33.593 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.594 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.594 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:33.594 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:33.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:34.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:35.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:37.562 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:37.593 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:37.594 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:37.594 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:37.594 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.595 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:37.596 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:38.563 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:39.565 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:40.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:41.608 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:41.609 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:41.609 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.610 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:41.611 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:41.611 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:42.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:43.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:44.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:45.611 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:45.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:45.612 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:45.613 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.614 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:45.614 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.614 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:45.614 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:45.614 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:46.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:47.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:48.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:49.614 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:49.615 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:49.615 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:49.615 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:49.615 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:49.615 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:49.615 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:49.616 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:49.616 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.616 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:49.616 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.616 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.616 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:49.617 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:49.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:50.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:51.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:52.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.616 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:53.617 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:53.618 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.619 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.619 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:53.619 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:53.968 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:58:53.957876', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:53.957876', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13676038583231394089, 'uuid_mslong': 7839128237629393148}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6cca2d99-40a2-48fc-bdcb-0ca1f80b3129'}, 'oper': 'CREATE', 'request-id': 'req-f2c45455-00f1-4b72-b900-1a1e9bad38ff', 'type': 'analytics_node', 'uuid': '6cca2d99-40a2-48fc-bdcb-0ca1f80b3129'} 09/01/2025 01:58:53.968 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:53.970 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.971 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7839128237629393148, "uuid_lslong": 13676038583231394089}, "enable": true, "created": "2025-09-01T01:58:53.957876", "last_modified": "2025-09-01T01:58:53.957876", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "6cca2d99-40a2-48fc-bdcb-0ca1f80b3129" } deleted = False >> 09/01/2025 01:58:53.971 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.989 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:58:53.973832', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:53.973832', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10557226880738800812, 'uuid_mslong': 4644487576518083042}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '407488d7-e213-41e2-9282-cde9d7a434ac'}, 'oper': 'CREATE', 'request-id': 'req-7052381b-b338-4995-aa45-2d649573240e', 'type': 'analytics_node', 'uuid': '407488d7-e213-41e2-9282-cde9d7a434ac'} 09/01/2025 01:58:53.990 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:53.992 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:53.992 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4644487576518083042, "uuid_lslong": 10557226880738800812}, "enable": true, "created": "2025-09-01T01:58:53.973832", "last_modified": "2025-09-01T01:58:53.973832", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "407488d7-e213-41e2-9282-cde9d7a434ac" } deleted = False >> 09/01/2025 01:58:53.992 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.082 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:58:54.061545', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:58:54.061545', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879227890580696674, 'uuid_mslong': 10756286823668662967}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '954601e0-7afc-42b7-96fa-c8203895ae62'}, 'oper': 'CREATE', 'request-id': 'req-c09260e8-71da-48cc-a61b-d3fea836d129', 'type': 'analytics_node', 'uuid': '954601e0-7afc-42b7-96fa-c8203895ae62'} 09/01/2025 01:58:54.083 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/01/2025 01:58:54.083 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.084 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10756286823668662967, "uuid_lslong": 10879227890580696674}, "enable": true, "created": "2025-09-01T01:58:54.061545", "last_modified": "2025-09-01T01:58:54.061545", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "954601e0-7afc-42b7-96fa-c8203895ae62" } deleted = False >> 09/01/2025 01:58:54.084 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:54.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:55.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:56.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:57.619 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:57.620 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:57.620 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:57.620 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:57.620 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:57.620 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:57.620 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:58:57.621 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:58.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:58:59.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:00.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:01.622 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:01.623 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:01.623 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:01.623 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:01.623 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:01.623 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:01.623 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:01.624 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:01.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:02.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:03.634 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:04.636 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:05.624 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:05.626 7ff0a4cceb80 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:05.626 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:05.626 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:05.626 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:05.626 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:05.626 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:05.627 7ff0a4cceb80 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:05.637 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:07.563 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:08.564 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.566 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:09.626 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:09.632 7ff0a4dc20e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:09.632 7ff0a4dc20e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:09.632 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:09.632 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:09.632 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:09.632 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:09.633 7ff0a4dc20e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:09.633 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.633 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:09.633 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.633 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.634 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:09.634 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.719 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:09.711039', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:09.711039', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410244128994570297, 'uuid_mslong': 7346777207241002013}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '65f4fef1-f189-4c1d-8297-e72d7e6d3039'}, 'oper': 'CREATE', 'request-id': 'req-65902cdc-006d-4333-bc45-edc4870831f6', 'type': 'analytics_alarm_node', 'uuid': '65f4fef1-f189-4c1d-8297-e72d7e6d3039'} 09/01/2025 01:59:09.720 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:09.720 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:09.721 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_alarm_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7346777207241002013, "uuid_lslong": 9410244128994570297}, "enable": true, "created": "2025-09-01T01:59:09.711039", "last_modified": "2025-09-01T01:59:09.711039", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "65f4fef1-f189-4c1d-8297-e72d7e6d3039" } deleted = False >> 09/01/2025 01:59:09.721 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.102 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:10.086800', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:10.086800', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10503918820682058891, 'uuid_mslong': 906375348849557938}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0c941793-dd59-41b2-91c5-6a83e84b4c8b'}, 'oper': 'CREATE', 'request-id': 'req-5937b566-f775-43d5-a1c9-186dda6980e6', 'type': 'analytics_alarm_node', 'uuid': '0c941793-dd59-41b2-91c5-6a83e84b4c8b'} 09/01/2025 01:59:10.102 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:10.103 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.103 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_alarm_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 906375348849557938, "uuid_lslong": 10503918820682058891}, "enable": true, "created": "2025-09-01T01:59:10.086800", "last_modified": "2025-09-01T01:59:10.086800", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "0c941793-dd59-41b2-91c5-6a83e84b4c8b" } deleted = False >> 09/01/2025 01:59:10.103 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.194 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:10.172514', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:10.172514', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879587110886434360, 'uuid_mslong': 11828816951027650397}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a4286658-5f1f-4f5d-96fc-0ed5b74de638'}, 'oper': 'CREATE', 'request-id': 'req-5363ae59-4436-4596-ac85-f2751a51eb9c', 'type': 'analytics_alarm_node', 'uuid': 'a4286658-5f1f-4f5d-96fc-0ed5b74de638'} 09/01/2025 01:59:10.194 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/01/2025 01:59:10.195 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.195 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_alarm_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11828816951027650397, "uuid_lslong": 10879587110886434360}, "enable": true, "created": "2025-09-01T01:59:10.172514", "last_modified": "2025-09-01T01:59:10.172514", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a4286658-5f1f-4f5d-96fc-0ed5b74de638" } deleted = False >> 09/01/2025 01:59:10.195 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:10.568 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:11.569 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:12.570 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:13.632 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:13.634 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.635 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:13.635 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.635 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:13.635 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:13.635 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:14.574 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:15.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:16.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:17.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:17.634 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:17.636 7ff0a4dc20e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:17.636 7ff0a4dc20e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.637 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:17.638 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.638 7ff0a4dc20e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:17.638 7ff0a4dc20e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:17.638 7ff0a4dc20e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:18.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:19.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:20.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:21.637 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:21.638 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.639 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:21.640 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.640 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.640 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:21.640 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.873 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.131', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-3.'], 'id_perms': {'created': '2025-09-01T01:59:21.867078', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:21.867078', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13422430401004267031, 'uuid_mslong': 671617895336398686}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'}, 'oper': 'CREATE', 'request-id': 'req-0bcb5258-1518-4af3-bc32-66c2afda7f6a', 'type': 'analytics_snmp_node', 'uuid': '095210ea-dc52-4f5e-ba46-0d55aa436617'} 09/01/2025 01:59:21.873 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:21.874 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:21.874 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-3. elements = { analytics_snmp_node_ip_address : "10.0.0.131"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 671617895336398686, "uuid_lslong": 13422430401004267031}, "enable": true, "created": "2025-09-01T01:59:21.867078", "last_modified": "2025-09-01T01:59:21.867078", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "095210ea-dc52-4f5e-ba46-0d55aa436617" } deleted = False >> 09/01/2025 01:59:21.874 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.233 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-2.'], 'id_perms': {'created': '2025-09-01T01:59:22.227040', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.227040', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10043001507123339914, 'uuid_mslong': 13054484782216790713}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'}, 'oper': 'CREATE', 'request-id': 'req-cc52ae74-0d4d-419c-a865-1eba05a95d49', 'type': 'analytics_snmp_node', 'uuid': 'b52ad8be-69c6-42b9-8b5f-e8a91ecca28a'} 09/01/2025 01:59:22.233 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:22.234 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.234 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. elements = { analytics_snmp_node_ip_address : "10.0.0.38"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13054484782216790713, "uuid_lslong": 10043001507123339914}, "enable": true, "created": "2025-09-01T01:59:22.227040", "last_modified": "2025-09-01T01:59:22.227040", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "b52ad8be-69c6-42b9-8b5f-e8a91ecca28a" } deleted = False >> 09/01/2025 01:59:22.234 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.316 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3995-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3995-1.'], 'id_perms': {'created': '2025-09-01T01:59:22.300301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:59:22.300301', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10295412184530401449, 'uuid_mslong': 11790706739729089604}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'}, 'oper': 'CREATE', 'request-id': 'req-79a7348d-975b-4e66-ac0e-d0c0e9ecf4b1', 'type': 'analytics_snmp_node', 'uuid': 'a3a1014f-585b-4444-8ee0-a6d59b29b8a9'} 09/01/2025 01:59:22.317 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/01/2025 01:59:22.317 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.317 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-1. elements = { analytics_snmp_node_ip_address : "10.0.0.249"display_name : "cn-jenkins-deploy-platform-ansible-os-3995-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-3995-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11790706739729089604, "uuid_lslong": 10295412184530401449}, "enable": true, "created": "2025-09-01T01:59:22.300301", "last_modified": "2025-09-01T01:59:22.300301", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a3a1014f-585b-4444-8ee0-a6d59b29b8a9" } deleted = False >> 09/01/2025 01:59:22.317 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:22.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:23.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:24.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:25.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:25.639 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:25.640 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:25.640 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:25.640 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:25.640 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:25.640 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:25.640 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:25.641 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:26.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:27.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:28.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:29.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:29.641 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:29.642 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.643 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:29.644 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:29.644 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:31.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:32.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:33.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:33.643 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:33.644 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.645 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:33.646 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.646 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.646 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:33.646 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:34.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:35.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:37.563 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:37.645 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:37.646 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:37.646 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:37.646 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:37.646 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:37.646 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:37.646 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:37.647 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:38.565 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:39.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:40.568 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.570 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:41.646 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:41.648 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:41.648 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:41.648 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:41.648 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:41.648 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:41.648 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:41.649 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:42.571 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:43.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:44.574 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:45.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:45.648 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:45.650 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:45.650 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:45.650 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:45.650 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:45.650 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:45.650 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:45.651 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:46.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:47.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:48.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:49.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:49.651 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:49.652 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:49.652 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:49.652 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:49.652 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:49.652 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:49.653 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:49.653 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:49.653 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.653 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:49.653 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.653 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:49.654 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:49.654 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:50.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:51.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:52.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:53.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:53.653 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:53.654 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:53.654 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:53.654 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:53.655 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:53.656 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:54.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:55.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:56.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:57.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:57.655 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:57.656 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:57.656 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.657 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:57.658 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 01:59:57.658 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 01:59:59.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:00.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:01.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:01.657 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:01.658 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:01.658 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:01.658 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:01.659 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.660 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:01.660 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:01.660 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:02.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:03.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:04.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:05.659 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:05.660 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:05.660 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:05.660 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:05.661 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:05.662 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.565 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:08.566 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:09.661 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:09.662 7ff0a4dc2720 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:09.662 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:09.662 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:09.663 7ff0a4dc2720 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:09.664 7ff0a4dc2720 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:10.568 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:11.570 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:12.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:13.663 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:13.664 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.665 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:13.665 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.665 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.665 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:13.665 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:14.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:15.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:16.579 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:17.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:17.664 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:17.665 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.666 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:17.667 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:17.667 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:18.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:19.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:20.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:21.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:21.667 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:21.669 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:21.670 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:21.670 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:21.670 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:21.670 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:21.670 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:21.671 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:22.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:23.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:24.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:25.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:25.670 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:25.671 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:25.671 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:25.671 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:25.671 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:25.672 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:26.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:27.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:28.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:29.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:29.671 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:29.673 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:29.673 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:29.673 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:29.673 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:29.673 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:29.673 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:29.674 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:30.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:31.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:32.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:33.674 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:33.675 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:33.675 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:33.675 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:33.675 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:33.676 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:34.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:35.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:37.565 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:37.675 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:37.677 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.678 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:37.678 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.678 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:37.678 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:37.678 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:38.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:39.569 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:40.571 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:41.677 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:41.679 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.680 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:41.680 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.680 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:41.680 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:41.680 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:42.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:43.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:44.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:45.579 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:45.679 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:45.681 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:45.682 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.682 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:45.682 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.682 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:45.682 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:45.682 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:46.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:47.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:48.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:49.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:49.682 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:49.683 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:49.683 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:49.683 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:49.683 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:49.683 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:49.683 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:49.684 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:50.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:51.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:52.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:53.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:53.683 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:53.685 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:53.686 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.686 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:53.686 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.686 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:53.686 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:53.686 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:55.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:56.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:57.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:57.685 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:57.687 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:57.687 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:57.687 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:57.687 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:57.687 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:57.687 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:00:57.688 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:00:59.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:00.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:01.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:01.687 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:01.688 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:01.688 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:01.688 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:01.689 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:03.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:04.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:05.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:05.689 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:05.691 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.692 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:05.692 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.692 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.692 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:05.692 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.567 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:08.569 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:09.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:09.692 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:09.693 7ff0a4dc2540 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:09.694 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.695 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:09.695 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.695 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:09.695 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:09.695 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.477 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-d6447694-7a63-4e16-8ab1-f2203b502079', 'type': 'global_system_config', 'uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8'} 09/01/2025 02:01:10.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:10.602 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2'], 'id_perms': {'created': '2025-09-01T02:01:10.595372', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.595372', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11952340681741785153, 'uuid_mslong': 6586627194885784908}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.192'}, 'oper': 'CREATE', 'request-id': 'req-681edb8a-0de7-491e-8459-bf06930acfe8', 'type': 'virtual_router', 'uuid': '5b686689-4b18-454c-a5df-3e8619ab5841'} 09/01/2025 02:01:10.602 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:01:10.603 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.603 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = an-jenkins-deploy-platform-ansible-os-3995-2 elements = { display_name : "an-jenkins-deploy-platform-ansible-os-3995-2"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6586627194885784908, "uuid_lslong": 11952340681741785153}, "enable": true, "created": "2025-09-01T02:01:10.595372", "last_modified": "2025-09-01T02:01:10.595372", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.192" } deleted = False >> 09/01/2025 02:01:10.603 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.941 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:01:10.925113', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:10.925113', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10735908003063543833, 'uuid_mslong': 17648289949859202616}}, 'parent_type': 'virtual-router', 'parent_uuid': '5b686689-4b18-454c-a5df-3e8619ab5841', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:f4:eb:52:34:6a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.941 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:01:10.942 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.942 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17648289949859202616, "uuid_lslong": 10735908003063543833}, "enable": true, "created": "2025-09-01T02:01:10.925113", "last_modified": "2025-09-01T02:01:10.925113", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "f4eb5234-6a74-4638-94fd-9b728fe03819"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:f4:eb:52:34:6a"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:01:10.942 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.967 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-1c3ccdca-0f5f-4532-9c6f-20034282c8ff', 'type': 'virtual_machine_interface', 'uuid': 'f4eb5234-6a74-4638-94fd-9b728fe03819'} 09/01/2025 02:01:10.973 7ff0a4dc2540 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.974 7ff0a4dc2540 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-2:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-2", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 17648289949859202616, "uuid_lslong": 10735908003063543833}, "enable": true, "created": "2025-09-01T02:01:10.925113", "last_modified": "2025-09-01T02:01:10.956445", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "5b686689-4b18-454c-a5df-3e8619ab5841"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7"}]uuid : "f4eb5234-6a74-4638-94fd-9b728fe03819"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:f4:eb:52:34:6a"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:01:10.974 7ff0a4dc2540 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.461 7ff0a4cce720 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = ec1d8a2b-6948-4c7d-ac90-b318244c2300 object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = http://10.0.0.38:8082/access-control-list/ec1d8a2b-6948-4c7d-ac90-b318244c2300 operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-3995-3:/usr/local/bin/contrail-schema remote_ip = 10.0.0.131 body = {"access-control-list": {"uuid": "ec1d8a2b-6948-4c7d-ac90-b318244c2300", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "access_control_list_hash": 905938936112430291}} domain = default-domain project = admin user = admin >> 09/01/2025 02:01:11.478 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-8b202558-6c4f-4e38-a46f-f0bfe3ec33be', 'type': 'access_control_list', 'uuid': 'ec1d8a2b-6948-4c7d-ac90-b318244c2300'} 09/01/2025 02:01:11.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:11.805 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:01:11.806 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:01:12.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:13.694 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:13.695 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:13.696 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.697 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:13.697 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.697 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:13.697 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:13.697 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:15.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:16.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:17.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:17.696 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:17.698 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.699 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:17.699 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.699 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:17.699 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:17.699 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:19.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:20.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:21.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:21.699 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:21.700 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:21.700 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:21.700 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:21.700 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:21.701 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:23.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:24.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:25.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:25.701 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:25.702 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:25.702 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:25.702 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:25.702 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:25.702 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:25.702 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:25.703 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:27.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:28.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:29.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:29.703 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:29.704 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:29.704 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:29.704 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:29.704 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:29.704 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:29.704 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:29.705 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:31.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:32.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:33.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:33.704 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:33.706 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:33.706 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:33.706 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:33.706 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:33.706 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:33.706 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:33.707 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:35.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:37.568 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:37.707 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:37.708 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:37.708 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:37.708 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:37.708 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:37.708 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:37.708 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:37.709 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.569 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:39.570 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:40.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:41.709 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:41.711 7ff0a4dc2c20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:41.711 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:41.711 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:41.711 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:41.711 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:41.711 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:41.712 7ff0a4dc2c20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:43.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:43.714 7ff0a4f9d2c0 [contrail-api] [ERROR]: Failed to resync: Unable to establish connection to http://10.0.0.249:5000/v3/domains? 09/01/2025 02:01:43.714 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.714 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:43.714 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:44.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:45.579 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:45.711 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:45.713 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:45.713 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:45.713 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:45.713 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:45.713 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:45.713 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:45.714 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:47.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:48.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:49.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:49.713 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:49.714 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:49.715 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.716 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:49.716 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.716 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:49.716 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:49.716 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:51.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:52.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:53.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:53.716 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:53.717 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:53.717 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:53.717 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:53.717 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:53.718 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:55.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:56.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:57.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:57.718 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:57.719 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:57.719 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:57.719 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:57.719 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:57.719 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:57.719 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:01:57.720 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:01:59.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:00.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:01.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:01.720 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:01.721 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:01.721 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:01.721 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:01.721 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:01.722 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:02.994 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3995-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1'], 'id_perms': {'created': '2025-09-01T02:02:02.985199', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:02.985199', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10627036196304171688, 'uuid_mslong': 16146534956283939804}}, 'parent_type': 'global-system-config', 'parent_uuid': 'dee9192c-e41b-4855-8333-2236491cfcf8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.18'}, 'oper': 'CREATE', 'request-id': 'req-7f54cc47-40c2-4259-85af-4adeb8e3ee0f', 'type': 'virtual_router', 'uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8'} 09/01/2025 02:02:02.995 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/01/2025 02:02:02.995 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.995 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = an-jenkins-deploy-platform-ansible-os-3995-1 elements = { display_name : "an-jenkins-deploy-platform-ansible-os-3995-1"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 16146534956283939804, "uuid_lslong": 10627036196304171688}, "enable": true, "created": "2025-09-01T02:02:02.985199", "last_modified": "2025-09-01T02:02:02.985199", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "dee9192c-e41b-4855-8333-2236491cfcf8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.20.0.18" } deleted = False >> 09/01/2025 02:02:02.995 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.332 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:02:03.316910', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:02:03.316910', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12779270811540185455, 'uuid_mslong': 492471125198063140}}, 'parent_type': 'virtual-router', 'parent_uuid': 'e01403fe-a6bb-4bdc-937a-d11db7edb2a8', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:06:d5:9b:e1:0f']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'a53b9506-b472-4351-b388-6abd2a36a2e7'}]}, 'oper': 'CREATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.332 7ff0a5a5e040 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/01/2025 02:02:03.333 7ff0a4c0d180 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.333 7ff0a4c0d180 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 492471125198063140, "uuid_lslong": 12779270811540185455}, "enable": true, "created": "2025-09-01T02:02:03.316910", "last_modified": "2025-09-01T02:02:03.316910", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "06d59be1-0f91-4624-b159-17187a5ee16f"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:06:d5:9b:e1:0f"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:02:03.333 7ff0a4c0d180 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.358 7ff0a5a5e040 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3995-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-e0b4c5ea-eae1-45cd-9751-832bb456eea9', 'type': 'virtual_machine_interface', 'uuid': '06d59be1-0f91-4624-b159-17187a5ee16f'} 09/01/2025 02:02:03.364 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.364 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:an-jenkins-deploy-platform-ansible-os-3995-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "an-jenkins-deploy-platform-ansible-os-3995-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "member", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 492471125198063140, "uuid_lslong": 12779270811540185455}, "enable": true, "created": "2025-09-01T02:02:03.316910", "last_modified": "2025-09-01T02:02:03.346223", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "e01403fe-a6bb-4bdc-937a-d11db7edb2a8"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "fa187b15-223c-4e06-9490-9a6decc1dbc7"}]uuid : "06d59be1-0f91-4624-b159-17187a5ee16f"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:06:d5:9b:e1:0f"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "a53b9506-b472-4351-b388-6abd2a36a2e7"}] } deleted = False >> 09/01/2025 02:02:03.364 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:04.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:05.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:05.722 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:05.725 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:05.725 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:05.725 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:05.725 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:05.725 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:05.725 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:05.726 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.570 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:08.571 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:09.725 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:09.727 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:09.727 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:09.727 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:09.727 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:09.727 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:09.728 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:09.729 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:11.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:12.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:13.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:13.728 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:13.730 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:13.730 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:13.730 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:13.730 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:13.731 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:15.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:16.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:17.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:17.731 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:17.733 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:17.733 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:17.733 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:17.733 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:17.733 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:17.733 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:17.734 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:19.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:20.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:21.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:21.734 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:21.735 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.736 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:21.737 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.737 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:21.737 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:21.737 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:23.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:24.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:25.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:25.736 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:25.738 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:25.738 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:25.738 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:25.738 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:25.738 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:25.738 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:25.739 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:27.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:28.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:29.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:29.739 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:29.741 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:29.741 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:29.741 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:29.741 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:29.741 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:29.741 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:29.742 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:31.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:32.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:33.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:33.741 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:33.743 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:33.743 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:33.743 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:33.743 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:33.744 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:35.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:37.570 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:37.744 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:37.745 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:37.745 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:37.745 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:37.745 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:37.745 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:37.745 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:37.746 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:39.574 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:40.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:41.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:41.745 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:41.747 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:41.748 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.748 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Down description = Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409 >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[Error: Unable to establish connection to http://10.0.0.249:5000/v3/domains? at UTC 2025-09-01 02:01:43.714409] connection down >>, ] >> 09/01/2025 02:02:41.748 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.748 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:41.748 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:41.748 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:43.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:44.040 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.040 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:44.040 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:45.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:45.748 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:45.749 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:45.749 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:45.749 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:45.749 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:45.749 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:45.750 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:47.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:48.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:49.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:49.749 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:49.750 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:49.750 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:49.751 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:49.752 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:51.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:52.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:53.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:53.751 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:53.752 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:53.752 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:53.752 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:53.753 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:53.754 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:55.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:56.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:57.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:57.753 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:57.754 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.755 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:57.756 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.756 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:57.756 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:02:57.756 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:02:59.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:00.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:01.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:01.755 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:01.756 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:01.756 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.757 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:01.758 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:01.758 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:03.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:04.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:05.757 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:05.759 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:05.759 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:05.759 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:05.759 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:05.759 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:05.759 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:05.760 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:07.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:08.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:09.759 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:09.761 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.762 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:09.762 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.762 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:09.762 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:09.762 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:11.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:12.579 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.374 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:03:13.375 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:03:13.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:13.762 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:13.763 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:13.763 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:13.763 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:13.765 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:13.765 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:15.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:16.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:17.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:17.764 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:17.765 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.766 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:17.767 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.767 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:17.767 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:17.767 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:19.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:20.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:21.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:21.767 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:21.768 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:21.768 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:21.768 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:21.768 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:21.768 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:21.768 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:21.769 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:23.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:24.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:25.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:25.769 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:25.770 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:25.770 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:25.770 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:25.770 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:25.770 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:25.771 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:27.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:28.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:29.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:29.770 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:29.772 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:29.772 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:29.772 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:29.772 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:29.772 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:29.772 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:29.773 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:31.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:32.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:33.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:33.773 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:33.774 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:33.775 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.776 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:33.776 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:33.776 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:35.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:37.572 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:37.776 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:37.777 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:37.777 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:37.777 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:37.778 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:37.779 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:39.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:40.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.579 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:41.778 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:41.779 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.781 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:41.781 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:43.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:44.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:45.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:45.780 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:45.782 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:45.782 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:45.782 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:45.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:47.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:48.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:49.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:49.783 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:49.784 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:49.784 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:49.784 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:49.784 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:49.784 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:49.784 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:49.785 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:51.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:52.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:53.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:53.785 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:53.786 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:53.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.788 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:53.788 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.788 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.788 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:53.788 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:55.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:56.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:57.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:57.787 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:57.788 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:57.790 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:03:57.790 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:03:59.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:00.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:01.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:01.789 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:01.791 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:01.792 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.792 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:01.792 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.792 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:01.792 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:01.792 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:03.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:04.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:05.792 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:05.794 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:05.794 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:05.794 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:05.794 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:05.794 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:05.794 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:05.795 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:07.573 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:08.574 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:09.794 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:09.796 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:09.796 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:09.796 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:09.796 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:09.797 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:09.797 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:09.797 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:09.797 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.798 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:09.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:09.798 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:09.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:11.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:12.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:13.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:13.797 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:13.797 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:13.797 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:13.798 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.256 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:04:14.257 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:04:14.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:15.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:16.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:17.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:17.798 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:17.800 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:17.801 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.801 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:17.801 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.801 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:17.801 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:17.801 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:19.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:20.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:21.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:21.801 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:21.802 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:21.802 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:21.802 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:21.803 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:23.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:24.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:25.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:25.803 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:25.804 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:25.804 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.805 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:25.806 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:27.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:28.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:29.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:29.805 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:29.806 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:29.806 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:29.806 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:29.806 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:29.806 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:29.807 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:31.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:32.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:33.807 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:33.808 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:33.808 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:33.808 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:33.808 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:33.808 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:33.808 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:33.809 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:35.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:37.574 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:37.809 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:37.810 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:37.810 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:37.810 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:37.810 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:37.810 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:37.810 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:37.811 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:39.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:40.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:41.810 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:41.812 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:41.812 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:41.812 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:41.812 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:41.813 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:43.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:44.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:45.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:45.813 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:45.814 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:45.814 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:45.814 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:45.815 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:47.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:48.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:49.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:49.815 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:49.817 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:49.817 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:49.817 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:49.817 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:49.817 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:49.817 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:49.818 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:51.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:52.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:53.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:53.817 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:53.819 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:53.819 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:53.819 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:53.819 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:53.819 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:53.819 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:53.820 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:55.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:56.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:57.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:57.820 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:57.821 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:57.821 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:57.821 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:57.821 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:57.821 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:57.821 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:04:57.822 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:04:59.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:00.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:01.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:01.822 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:01.823 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:01.824 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.825 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:01.825 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:01.825 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:03.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:04.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:05.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:05.824 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:05.825 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:05.826 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.827 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:05.827 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.827 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:05.827 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:05.827 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.575 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:08.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:09.826 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:09.827 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.828 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:09.829 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.829 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:09.829 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:09.829 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:11.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:12.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:13.828 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:13.829 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.830 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:13.831 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:13.831 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:15.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:16.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:17.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:17.831 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:17.832 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:17.833 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.834 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:17.834 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.834 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:17.834 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:17.834 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:19.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:20.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:21.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:21.833 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:21.834 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.835 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:21.836 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:21.836 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:23.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:24.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:25.835 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:25.836 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:25.836 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:25.836 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:25.836 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:25.836 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:25.836 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:25.837 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:27.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:28.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:29.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:29.837 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:29.839 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:29.839 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:29.839 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:29.839 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:29.840 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:29.840 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:29.840 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:29.840 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.840 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:29.840 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.841 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.841 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:29.841 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:31.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:32.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:33.840 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:33.841 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:33.841 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:33.841 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:33.841 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:33.841 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:33.842 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:35.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:37.576 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:37.842 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:37.843 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:37.843 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:37.843 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:37.843 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:37.843 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:37.844 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:39.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:40.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:41.843 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:41.845 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:41.845 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:41.845 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:41.845 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:41.845 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:41.845 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:41.846 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:43.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:44.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:45.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:45.845 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:45.847 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:45.847 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:45.847 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:45.847 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:45.847 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:45.847 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:45.848 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:47.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:48.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:49.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:49.848 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:49.849 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:49.849 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:49.849 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:49.850 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:51.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:52.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:53.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:53.850 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:53.852 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:53.852 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:53.852 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:53.852 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:53.852 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:53.852 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:53.853 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:55.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:56.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:57.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:57.853 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:57.854 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.855 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:57.856 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:05:57.856 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:05:59.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:00.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:01.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:01.855 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:01.856 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.857 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:01.858 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.858 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:01.858 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:01.858 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:03.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:04.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:05.857 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:05.858 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:05.858 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:05.858 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:05.858 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:05.859 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.577 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:08.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:09.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:09.859 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:09.861 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:09.862 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.862 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:09.862 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.862 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:09.862 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:09.862 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:11.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:12.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:13.861 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:13.862 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.863 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:13.864 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:13.864 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:15.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:15.871 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:06:15.871 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:06:15.871 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:06:15.871 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:06:15.872 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:06:15.872 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:06:15.872 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:06:15.872 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:06:16.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:17.863 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:17.865 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.866 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:17.866 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.866 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.866 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:17.866 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:19.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:20.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:21.866 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:21.867 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:21.867 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:21.867 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:21.867 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:21.867 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:21.867 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:21.868 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:23.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:24.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:25.868 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:25.869 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:25.869 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:25.869 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:25.870 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.871 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.871 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:25.871 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:27.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:28.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:29.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:29.870 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:29.871 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:29.871 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:29.871 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:29.871 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:29.871 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:29.871 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:29.872 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:31.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:32.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:33.873 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:33.876 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:33.876 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:33.876 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:33.877 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.878 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:33.878 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:33.878 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:35.628 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:37.578 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:37.877 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:37.878 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:37.878 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:37.879 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:39.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:40.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:41.879 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:41.880 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:41.880 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:41.880 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:41.881 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:41.882 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:41.882 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:43.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:44.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:45.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:45.881 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:45.883 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:45.883 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:45.883 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:45.883 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:45.883 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:45.883 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:45.884 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:47.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:48.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:49.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:49.883 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:49.885 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.886 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:49.886 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.886 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:49.886 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:49.886 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:51.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:52.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:53.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:53.885 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:53.886 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:53.887 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:53.888 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:53.888 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:53.888 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:55.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:56.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:57.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:57.887 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:57.889 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:57.889 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:57.889 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:57.889 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:57.889 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:57.889 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:06:57.890 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:06:59.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:00.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:01.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:01.889 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:01.891 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:01.892 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.892 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:01.892 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.892 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.892 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:01.892 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:03.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:04.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:05.628 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:05.891 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:05.892 7ff0a4c0d220 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:05.892 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:05.892 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:05.892 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:05.893 7ff0a4c0d220 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:08.581 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:09.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:09.893 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:09.894 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:09.895 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:09.896 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:09.896 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:09.896 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:09.896 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:11.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:12.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:13.895 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:13.896 7ff0a4dc27c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:13.897 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:13.898 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:13.898 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:13.898 7ff0a4dc27c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:13.898 7ff0a4dc27c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:13.898 7ff0a4dc27c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:15.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:07:16.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:07:17.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:17.897 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:17.899 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:17.900 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:17.900 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:17.900 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:17.900 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:17.900 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:17.900 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:19.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:20.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:21.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:21.899 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:21.901 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:21.901 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:21.901 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:21.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:23.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:24.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:25.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:25.902 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:25.903 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:25.903 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:25.903 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:25.903 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:25.903 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:25.903 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:25.904 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:27.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:28.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:29.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:29.904 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:29.906 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:29.906 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:29.906 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:29.906 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:29.906 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:29.906 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:29.907 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:31.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:32.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:33.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:33.906 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:33.908 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:33.908 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:33.908 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:33.908 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:33.909 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:33.909 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:33.909 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:33.909 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:33.909 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:33.910 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:33.910 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:33.910 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:33.910 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:35.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:37.580 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:37.909 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:37.910 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:37.910 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:37.910 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:37.911 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:39.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:40.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:41.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:41.910 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:41.912 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:41.912 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:41.912 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:41.912 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:41.912 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:41.912 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:41.913 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:43.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:44.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:45.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:45.914 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:45.916 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:45.916 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:45.916 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:45.916 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:45.916 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:45.916 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:45.917 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:47.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:48.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:49.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:49.917 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:49.918 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:49.918 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:49.918 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:49.919 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:49.920 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:49.920 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:51.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:52.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:53.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:53.919 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:53.921 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:53.921 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:53.921 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:53.921 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:53.921 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:53.921 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:53.922 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:55.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:56.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:57.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:57.922 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:57.923 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:57.923 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:57.923 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:57.923 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:57.923 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:07:57.924 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:07:59.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:00.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:01.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:01.924 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:01.925 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:01.925 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:01.925 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:01.925 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:01.925 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:01.925 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:01.926 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:03.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:04.628 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:05.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:05.926 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:05.927 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:05.928 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:05.929 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:05.929 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:05.929 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:05.929 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:08.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:09.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:09.928 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:09.929 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:09.930 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:09.931 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:09.931 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:11.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:12.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:13.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:13.930 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:13.932 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:13.932 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:13.932 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:13.933 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:15.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:16.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:17.933 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:17.935 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:17.936 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:17.936 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:17.936 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:17.936 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:19.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:20.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:21.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:21.935 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:21.937 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:21.937 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:21.937 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:21.938 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:23.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:24.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:25.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:25.938 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:25.939 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:25.939 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:25.939 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:25.940 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:27.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:28.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:29.940 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:29.942 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:29.942 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:29.942 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:29.942 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:29.942 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:29.942 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:29.943 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:31.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:32.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:33.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:33.943 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:33.945 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:33.945 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:33.945 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:33.945 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:33.945 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:33.945 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:33.946 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:35.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:37.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:37.946 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:37.947 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:37.947 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:37.947 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:37.947 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:37.947 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:37.948 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.583 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:39.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:40.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:41.948 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:41.950 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:41.951 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:41.951 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:41.951 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:41.951 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:41.951 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:43.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:44.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:45.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:45.951 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:45.952 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:45.952 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:45.953 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:45.954 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:45.954 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:47.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:48.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:49.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:49.954 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:49.955 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:49.955 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:49.955 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:49.955 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:49.956 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:51.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:52.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:53.956 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:53.958 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:53.959 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:53.959 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:53.959 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:53.959 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:53.959 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:55.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:56.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:57.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:57.959 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:57.960 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:57.961 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:57.961 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:57.961 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:57.961 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:57.961 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:57.961 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:57.962 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:57.962 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:57.962 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:57.962 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:57.962 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:08:57.962 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:08:59.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:00.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:01.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:01.961 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:01.963 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:01.963 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:01.963 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:01.964 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:01.965 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:01.965 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:01.965 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:03.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:04.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:05.963 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:05.965 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:05.965 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:05.965 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:05.965 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:05.965 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:05.965 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:05.966 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.582 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:08.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:09.966 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:09.967 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:09.967 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:09.967 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:09.968 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:09.969 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:11.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:12.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:13.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:13.969 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:13.970 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:13.970 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:13.970 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:13.970 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:13.970 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:13.970 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:13.971 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:15.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:16.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:17.971 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:17.973 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:17.973 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:17.973 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:17.973 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:17.973 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:17.973 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:17.974 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.365 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:09:18.365 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:09:18.366 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:09:18.366 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:09:18.366 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:09:18.366 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:09:18.366 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:09:18.366 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:09:18.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:19.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:20.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:21.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:21.973 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:21.975 7ff0a4c0d2c0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:21.975 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:21.975 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:21.975 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:21.975 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:21.975 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:21.976 7ff0a4c0d2c0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:23.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:24.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:25.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:25.976 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:25.977 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:25.977 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:25.977 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:25.977 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:25.977 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:25.977 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:25.978 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:27.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:28.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:29.978 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:29.979 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:29.979 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:29.979 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:29.979 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:29.980 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:31.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:32.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:33.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:33.980 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:33.982 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:33.983 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:33.983 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:33.983 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:33.983 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:33.983 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:35.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:37.584 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:37.983 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:37.984 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:37.984 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:37.984 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:37.984 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:37.984 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:37.985 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:39.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:40.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:41.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:41.985 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:41.987 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:41.987 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:41.987 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:41.987 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:41.987 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:41.987 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:41.988 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:43.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:44.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:45.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:45.988 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:45.990 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:45.990 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:45.990 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:45.990 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:45.990 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:45.990 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:45.991 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:47.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:48.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:49.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:49.990 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:49.991 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:49.991 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:49.992 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:49.993 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:49.993 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:51.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:52.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:53.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:53.992 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:53.993 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:53.993 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:53.993 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:53.994 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:55.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:56.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:57.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:57.994 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:57.995 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:57.995 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:57.995 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:09:57.996 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:09:59.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:00.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:01.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:01.996 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:01.998 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:01.998 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:01.998 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:01.999 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:01.999 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:01.999 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:01.999 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:02.000 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.000 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:02.000 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.000 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.000 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:02.000 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:03.628 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:04.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:05.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:05.998 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:06.000 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:06.000 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:06.000 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.000 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:06.000 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.000 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:06.001 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.585 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:08.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:09.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:10.001 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:10.002 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:10.002 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:10.002 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.002 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:10.002 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:10.003 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:11.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:12.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:13.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:14.003 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:14.004 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:14.004 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:14.004 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.004 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:14.004 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.004 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:14.005 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:15.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:16.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:17.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:18.004 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:18.006 7ff0a4c0d400 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:18.006 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:18.006 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.006 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:18.006 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.006 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:18.007 7ff0a4c0d400 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:19.236 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:10:19.237 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:10:19.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:20.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:21.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:22.007 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:22.009 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:22.009 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:22.010 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.010 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:22.010 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.010 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:22.011 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:23.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:24.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:25.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:26.010 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:26.011 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:26.012 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.013 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:26.013 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.013 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.013 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:26.013 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:27.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:28.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:29.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:30.013 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:30.014 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:30.014 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:30.014 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.014 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:30.014 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.014 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:30.015 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:31.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:32.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:33.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:34.015 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:34.016 7ff0a4dc24a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:34.016 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:34.016 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.016 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:34.016 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.016 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:34.017 7ff0a4dc24a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:35.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:37.586 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:38.017 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:38.018 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:38.018 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:38.018 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.018 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:38.018 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.018 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:38.019 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:39.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:40.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:41.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:42.018 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:42.020 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:42.021 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.021 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:42.021 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.021 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.021 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:42.021 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:43.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:44.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:45.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:46.021 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:46.023 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:46.023 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:46.023 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.023 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:46.023 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.023 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:46.024 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:46.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:47.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:48.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:49.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:50.024 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:50.025 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:50.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.027 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:50.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:50.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:51.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:52.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:53.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:54.026 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:54.027 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:54.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:54.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.027 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:54.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:54.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:54.028 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:55.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:56.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:57.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:58.027 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:58.029 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:58.029 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:58.029 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.029 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:58.029 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:58.029 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:10:58.030 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:58.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:10:59.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:00.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:01.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:02.030 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:02.031 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:02.031 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:02.031 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.032 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.033 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:02.033 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:02.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:03.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:04.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:05.636 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:06.031 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:06.033 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:06.033 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:06.033 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.033 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:06.033 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:06.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:08.588 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:09.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:10.034 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:10.036 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:10.036 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:10.036 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.036 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:10.036 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:10.036 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:10.037 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:10.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:11.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:12.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:13.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:14.037 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:14.038 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:14.038 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:14.038 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.038 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:14.038 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:14.038 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:14.039 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:14.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:15.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:16.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:17.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:18.039 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:18.040 7ff0a4c0d4a0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:18.040 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.041 7ff0a4c0d4a0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:18.042 7ff0a4c0d4a0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:18.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:19.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:20.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:21.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:22.041 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:22.043 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.044 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:22.044 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.044 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.044 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:22.044 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:22.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:23.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:24.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:25.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:26.044 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:26.045 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:26.045 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:26.045 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.045 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:26.045 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:26.046 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:26.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:27.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:28.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:29.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:30.046 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:30.048 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:30.048 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:30.048 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.049 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.050 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:30.050 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:30.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:31.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:32.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:33.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:34.049 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:34.051 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:34.051 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:34.051 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.051 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:34.051 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:34.051 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:34.052 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:34.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:35.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:37.587 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:38.051 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:38.053 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:38.053 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:38.053 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.053 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:38.053 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:38.054 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:38.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:39.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:40.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:41.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:42.054 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:42.056 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:42.057 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.057 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:42.057 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.057 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.057 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:42.057 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:42.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:43.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:44.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:45.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:46.056 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:46.058 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:46.058 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:46.058 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.058 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:46.058 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:46.058 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:46.059 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:46.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:47.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:48.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:49.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:50.059 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:50.061 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:50.061 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:50.061 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.061 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:50.061 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:50.062 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:50.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:51.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:52.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:53.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:54.061 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:54.063 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:54.063 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:54.063 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.063 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:54.063 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:54.063 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:54.064 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:55.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:56.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:57.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:58.064 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:58.065 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:58.065 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.066 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:11:58.067 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:58.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:11:59.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:00.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:01.634 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:02.067 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:02.068 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:02.068 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:02.068 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.068 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:02.069 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:02.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:03.637 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:04.639 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:05.640 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:06.069 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:06.071 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:06.071 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:06.071 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.071 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:06.071 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:06.071 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:06.072 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:08.590 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:09.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:10.072 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:10.073 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:10.074 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.075 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.075 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:10.075 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:10.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:11.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:12.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:13.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:14.074 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:14.075 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:14.075 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:14.075 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:14.076 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:14.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:15.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:16.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:17.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:18.076 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:18.078 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:18.078 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:18.078 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.078 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:18.078 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:18.078 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:18.079 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:18.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:19.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:12:20.870 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:12:21.612 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:22.079 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:22.081 7ff0a4c0d360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:22.081 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:22.081 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.081 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:22.081 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:22.081 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:22.082 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:22.082 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.082 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:22.082 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.082 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.083 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:22.083 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:22.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:23.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:24.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:25.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:26.082 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:26.083 7ff0a4c0d360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:26.083 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:26.083 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:26.084 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:26.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:27.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:28.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:29.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:30.084 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:30.086 7ff0a4c0d360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.087 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:30.088 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.088 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.088 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:30.088 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:30.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:31.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:32.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:33.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:34.087 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:34.088 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.089 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.090 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:34.090 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:34.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:35.636 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:37.589 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:38.089 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:38.090 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:38.090 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:38.090 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.090 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:38.090 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:38.090 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:38.091 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:38.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:39.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:40.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:41.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:42.090 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.092 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:42.093 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.093 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.093 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:42.093 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:42.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:43.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:44.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:45.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:46.093 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:46.094 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:46.094 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:46.094 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:46.095 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:46.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:47.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:48.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:49.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:50.095 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:50.096 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:50.096 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:50.096 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:50.097 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:50.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:51.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:52.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:53.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:54.097 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:54.099 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:54.100 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.100 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:54.100 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.100 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.100 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:54.100 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:55.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:56.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:57.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:58.100 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:58.102 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:58.103 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.103 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:58.103 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.103 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.103 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:12:58.103 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:58.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:12:59.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:00.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:01.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:02.103 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:02.104 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:02.104 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:02.104 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.104 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:02.104 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:02.104 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:02.105 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:02.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:03.634 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:04.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:05.637 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:06.105 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:06.107 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:06.107 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:06.107 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.107 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:06.107 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:06.107 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:06.108 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.591 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:08.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:09.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:10.108 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:10.109 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:10.109 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:10.109 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.110 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:10.110 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:10.110 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:10.110 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:10.110 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.111 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:10.111 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.111 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.111 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:10.111 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:10.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:11.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:12.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:13.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:14.111 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:14.113 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:14.113 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:14.113 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.113 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:14.113 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:14.113 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:14.114 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:14.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:15.605 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:16.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:17.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:18.113 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:18.115 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:18.115 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:18.115 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.115 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:18.116 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:18.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:19.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:20.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:13:21.758 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:13:22.116 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:22.118 7ff0a4c0d360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:22.118 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:22.118 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.118 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:22.118 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:22.118 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:22.119 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:22.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:23.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:24.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:25.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:26.119 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:26.120 7ff0a4c0d360 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:26.120 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:26.120 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.120 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:26.120 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:26.120 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:26.121 7ff0a4c0d360 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:26.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:27.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:28.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:29.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:30.121 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:30.123 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:30.123 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:30.123 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.123 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:30.123 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:30.123 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:30.124 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:30.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:31.630 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:32.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:33.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:34.124 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:34.125 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:34.125 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.126 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.127 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:34.127 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:34.634 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:35.636 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:37.592 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:38.126 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:38.127 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:38.127 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:38.127 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.128 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:38.129 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:38.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:39.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:40.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:41.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:42.128 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:42.130 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:42.130 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:42.130 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.130 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:42.130 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:42.130 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:42.131 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:42.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:43.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:44.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:45.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:46.130 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:46.132 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:46.132 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:46.132 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.132 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:46.132 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:46.132 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:46.133 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:46.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:47.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:48.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:49.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:50.133 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:50.134 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:50.134 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:50.134 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.134 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:50.134 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:50.134 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:50.135 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:50.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:51.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:52.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:53.618 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:54.135 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:54.136 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:54.136 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:54.136 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.136 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.137 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:54.138 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:55.623 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:56.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:57.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:58.137 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:58.138 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:58.138 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.139 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:13:58.140 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:58.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:13:59.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:00.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:01.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:02.140 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:02.141 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:02.141 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:02.142 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.143 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.143 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:02.143 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:02.635 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:03.636 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:04.638 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:05.640 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:06.142 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:06.144 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:06.144 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:06.144 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.144 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:06.145 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:08.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:09.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:10.145 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:10.146 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:10.146 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:10.146 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.147 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.148 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:10.148 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:10.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:11.600 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:12.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:13.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:14.147 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:14.149 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:14.150 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.150 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:14.150 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.150 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.150 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:14.150 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:14.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:15.608 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:16.610 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:17.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:18.149 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:18.151 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:18.151 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:18.151 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.151 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:18.151 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:18.151 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:18.152 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:18.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:19.615 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:20.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:21.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:22.152 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:22.153 7ff0a4dc2f40 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:22.153 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:22.153 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:22.154 7ff0a4dc2f40 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:22.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:23.621 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:24.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:25.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:26.154 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:26.155 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:26.155 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.156 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:26.157 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:26.625 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:27.628 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:28.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:29.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:30.157 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:30.158 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:30.158 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.159 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:30.160 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.160 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.160 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:30.160 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:30.633 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:31.634 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:32.636 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:33.638 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:34.161 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:34.163 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.164 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:34.164 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.164 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.164 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:34.164 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:34.640 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:35.641 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:37.593 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:38.163 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:38.164 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:38.165 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.166 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:38.166 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.166 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.166 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:38.166 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:38.595 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:39.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:40.597 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:41.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:42.165 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:42.167 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:42.167 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:42.168 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.169 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.169 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:42.169 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:42.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:43.602 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:44.604 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:45.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:46.168 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:46.169 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:46.169 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.170 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.171 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:46.171 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:46.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:47.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:48.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:49.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:50.170 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:50.172 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:50.172 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:50.172 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:50.172 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:50.172 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:50.172 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:50.173 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:50.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:51.616 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:52.617 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:53.619 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:54.172 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:54.174 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:54.174 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:54.174 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:54.174 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:54.175 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.620 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:55.622 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:56.624 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:57.626 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:58.176 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:58.177 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:58.178 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:58.178 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:58.178 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:58.178 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:58.178 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:58.178 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:58.179 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.179 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:58.179 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.179 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.179 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:14:58.179 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.627 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:14:59.629 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:00.631 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:01.632 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:02.178 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:02.180 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:02.181 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.181 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:02.181 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.181 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.181 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:02.181 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:02.634 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:03.637 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:04.639 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:05.641 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:06.180 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:06.181 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:06.181 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.182 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.183 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:06.183 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.594 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:08.596 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:09.598 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:10.183 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:10.184 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:10.184 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:10.184 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.185 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:10.186 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:10.599 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:11.601 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:12.603 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:13.606 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:14.186 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:14.187 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:14.187 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:14.187 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:14.187 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:14.187 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.188 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:14.189 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:14.607 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:15.609 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:16.611 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:17.613 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:18.188 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:18.189 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:18.189 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:18.189 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:18.190 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:18.614 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:19.641 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:20.317 7ff0a6fa6d60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/01/2025 02:15:20.320 7ff0a6fa6d60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:15:20.642 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:21.644 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:22.190 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:22.191 7ff0a6fa6d60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:22.191 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:22.191 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:22.191 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:22.191 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:22.191 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.249:2181, 10.0.0.38:2181, 10.0.0.131:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.131:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.249:9041, 10.0.0.38:9041, 10.0.0.131:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.249:5673, 10.0.0.38:5673, 10.0.0.131:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.249:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3995-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1756691825542814 collector_name = collector_ip = 10.0.0.131:8086 collector_list = [ 10.0.0.131:8086, 10.0.0.38:8086, 10.0.0.249:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/01/2025 02:15:22.192 7ff0a6fa6d60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:22.646 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 09/01/2025 02:15:23.346 7ff0a4dc2540 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 09/01/2025 02:15:23.647 7ff0a4cce0e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.